Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to the claims amendment filed on 07/31/2025.
Claims 1-20 are presented for examination.
Response to Argument
In reference to applicant’s argument regrading rejections under 35 U.S.C. § 101:
Applicant’s Argument:
The applicant’s argument regarding the 101 rejection based on the claim amendment filed on 07/31/2025.
Examiner’s Response:
The 101 rejection is withdrawn in view of the claim amendment filed on 07/31/2025.
In reference to applicant’s argument regrading rejections under 35 U.S.C. § 103:
Applicant’s Argument:
Applicant’s argument regarding the 103 rejection based on the current claim amendment filed on 07/31/2025.
Examiner’s Response:
This argument includes the newly amended limitations. It has been fully considered but is moot in view of the new grounds of rejection presented below necessitated by the amendment.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims1-20 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. There does not seem to be sufficient description for:
The claim 1 recites “receive, from one or more client devices, at least one request to train a first machine- learning model, the at least one request comprising a first data identifier, a first set of transformation parameters, and a first set of hyperparameters; based on the at least one request, train the first machine-learning model utilizing a pre- defined framework of computer-executable instructions having a plurality of template code bases comprising a data generation model template code base, a data transformation model template code base”, “receive, from at least one client device, at least one additional request to train a second machine-learning model, the at least one additional request comprising a second data identifier, a second set of transformation parameters, and a second set of hyperparameters; and based on the at least one additional request, train the second machine-learning model utilizing the pre-defined framework of computer-executable instructions and the second data identifier, the second set of transformation parameters, and the second set of hyperparameters.” at lines 3-10, and lines 23-27. However, the specification [Par.0066], “The data generation model container 308 can receive the data identifier 324 in one or more different ways. In certain embodiments, executing the code of the data generation model container 308 causes the machine-learning platform system 102 to instantiate a graphical user interface on the model engineering device(s) 302 requesting the data identifier 324. For instance, the graphical user interface may include an interactive prompt, a screen alert, or a digital notification requesting user input that corresponds to the data identifier 324. In other embodiments, the data generation model container 308 comprises place-holder coding, pseudocode, or commented-out portions indicating a request for a user to enter in the data identifier 324.” [0067] Upon execution, the data generation model container 308 uses the data identifier 324 to generate digital queries 326 (e.g., information retrieval requests) and [Par.0154], “The computing device 600 also includes one or more input or output interface 608 (or "I/O interface 608"), which are provided to allow a user (e.g., requester or provider) to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device 600.” The specification only describes the input interface that allow the user to input/request but it does not describe the request to train the machine learning model. Therefore, the claim 1 is failing to comply with the written description requirement.
The claims 2-9 are defendant of the claim 1 and are likewise failing to comply with the written description requirement.
The claim 10 is being rejected as the same reason as the claim 1.
The claims 11-16 are defendant of the claim 10 and are likewise failing to comply with the written description requirement.
Regarding claim 17 is rejected for the same reason as the claim 1, since these claims recite the same limitations.
The claims 18-20 are defendant of the claim 17 and are likewise failing to comply with the written description requirement.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 6-13, 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over McNee et al. (Pub. No. US 20220150275-hereinafter, McNee) and further in view of YU et al. (PUB. No. US 20180121814 -hereinafter, Yu) and further in view of Bendre et al. (PUB. No. US. 20180322417-Hereinafter, Bendre).
Regarding claim 1, McNee teachesA non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause a computing device to (McNee, [Par.0089], “Furthermore, in some embodiments, some or all of the components of the EPSS 1310 may be implemented or provided in other manners, such as at least partially in firmware and/or hardware, including, but not limited to one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers executing appropriate instructions, and including microcontrollers and/or embedded controllers, field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and the like. Some or all of the system components and/or data structures may also be stored as contents (e.g., as executable or other machine-readable software instructions or structured data) on a computer-readable medium (e.g., a hard disk; memory; network; other computer-readable medium; or other portable media article to be read by an appropriate drive or via an appropriate connection, such as a DVD or flash memory device) to enable the computer-readable medium to execute or otherwise use or provide the contents to perform at least some of the described techniques. Some or all of the components and/or data structures may be stored on tangible, non-transitory storage mediums. Some or all of the system components and data structures may also be stored as data signals (e.g., by being encoded as part of a carrier wave or included as part of an analog or digital propagated signal) on a variety of computer-readable transmission mediums, which are then transmitted, including across wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments.” And [Par.0090], “Some or all of the components and/or data structures may be stored on tangible, non-transitory storage mediums. Some or all of the system components and data structures may also be stored as data signals (e.g., by being encoded as part of a carrier wave or included as part of an analog or digital propagated signal) on a variety of computer-readable transmission mediums, which are then transmitted, including across wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of this disclosure may be practiced with other computer system configurations.”):
and a training-tuning model template code base, by extracting a set of data utilizing the data generation model template code base(McNee, [Par.0036-0040], “[0031], Depending upon the machine learning environment, some portions of this pipeline may be facilitated by human interaction. In the EPSS configurations described herein, the framework for building and tuning new models facilitates and makes more repeatable and efficient the generation of acceptable models 220. Some portions of this process can be automated using this framework such as trying a series of different tuning parameters using autogenerated models created from metadata stored in the trained model library 117 of FIG. 1. [0036]Predictive Security System. The logic described in FIG. 4 describes use of the EPSS framework to generate and/or tune predictive cybersecurity threat analysis applications for deployment. Portions of this logic may be optional in some predictive security systems and different logic may be executed in a loop to tune applications once deployed. [0037] In block 401, the system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information (e.g., administrator and ownership information), IP (internet protocol) addresses, DNS record data, passive DNS activity data, scraped HTML content, TLS certificate information, and/or other domain related data. This data may be collected from a variety of sources and at different cadences and may be generated by the EPSS itself. For example, blocklist data which indicates known malicious domains, are available from a variety of services which update typically at least daily. For example, such data is available from organizations or companies such as The Spamhaus Project, an international organization that delivers lists of blocked IP addresses and blocked domains as soon as they are researched and added to their threat lists. Other private companies and other organizations provide similar data or subsets of such data. Other types of IID may be updated once a day or less frequently, for example some are streamed in near real-time, others are forwarded weekly, bi-weekly, monthly, etc. For example, public DNS (“A” record) data are available to all DNS servers participating in internet traffic, as they are the “directory” entries that map top-level logical names (such as “domaintools.com”) to IP addresses. Passive DNS activity data are packets that indicate that at some point in time a domain has been associated with a specific DNS record. This data is collected and distributed, for example, by a service such as a hosting company or other Internet Service Provider (ISP)… feature engineering is performed to engineer feature vectors for use in machine learning models. In overview, a feature vector is used as a way to select (extract, determine, or the like) and encode domain information for input into a machine learning algorithm. For example, a feature vector applied to a type of domain related data stored in the EPSS domain table (such as “domain name”), can be used to encode information about each domain based upon whether the domain matches or doesn't match specific criteria.” Examiner’s note, domain data is extracted based on the metadata of the EPSS and the domain identifier (data identifier) provided by the client devices (hosting company or other Internet Service Provider (ISP). The domain information is encoded for input into the machine learning model, that is corresponding to the model template code.).
generating a training dataset utilizing the data transformation model template code base of the pre-defined framework of computer-executable instructions(Mcnee, [Par.0020-0026], “0020, [Par.0020], “In one powerful incarnation, in overview the EPSS uses a domain centric approach combined with advanced machine learning algorithms and a multi-level machine learning architecture that utilizes one or more subsets of the smaller models trained with different data, whose results are combined as input to an (at least one) ensemble master classifier, which can be ultimately tuned and optimized for the type of data it is responsible for classifying. Each subset of the smaller models includes multiple instances of a same model sharing a same machine learning algorithm, modeling tuning parameters, and feature vector values but trained using different trained data. Hence each model subset acts as a set of “weak classifiers” for a particular type or collection of threat data. Certain subsets may be more applicable and tuned for certain types of applications because they pull (access, assess, determine, etc.) different domain related data, or are tuned differently, etc. A combination of the results of each applicable subset of weak classifiers then is fed as input into the ensemble master classifier, which can be iteratively run with varying weights applied to the weak classifier subset outputs until a determined optimization value (e.g., a threshold, minimum, percentage, probability, etc.) is reached. The resultant ensemble master classifier can then be deployed as a cybersecurity threat analysis application applied to an unknown domain to predict whether it is “predictably malicious.”…FIG. 1 is a block diagram of components of an example Enhanced Predictive Security System described herein. In one example embodiment, the Enhanced Predictive Security System comprises one or more functional components/modules that work together to provide an architecture and a framework for building and deploying cybersecurity threat analysis application. For example, the EPSS 100 may comprise one or more machine learning algorithms 110, feature class engines (for use with feature engineering) 111, tuning systems 112, ensemble classifier engines 113, and validation and testing engines 114. These components cooperate and act upon domain data and feature class vectors (stored in a repository 115), to create sampled test, training, and validation data 116 and to build model subsets and applications using trained model library 117. In an example EPSS configuration, the trained model library 117 stores definitions of each model subset for easy re-instantiation, including an indication of the machine learning algorithm used to create the model along with hyper parameters for tuning the model, and a description of the feature class information used to build an input feature vector associated with the model, an indication of a source for training data, and an indication of training data sampling parameters. Other versions of the model library 117 may contain more or less or different information…” and [0037-0040] In block 401, the system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information (e.g., administrator and ownership information), IP (internet protocol) addresses, DNS record data, passive DNS activity data, scraped HTML content, TLS certificate information, and/or other domain related data. This data may be collected from a variety of sources and at different cadences and may be generated by the EPSS itself. For example, blocklist data which indicates known malicious domains, are available from a variety of services which update typically at least daily. For example, such data is available from organizations or companies such as The Spamhaus Project, an international organization that delivers lists of blocked IP addresses and blocked domains as soon as they are researched and added to their threat lists…This data is collected and distributed, for example, by a service such as a hosting company or other Internet Service Provider (ISP)… For example, a feature vector applied to a type of domain related data stored in the EPSS domain table (such as “domain name”), can be used to encode information about each domain based upon whether the domain matches or doesn't match specific criteria.” Examiner’s note, the training data is created/generated based on feature class vector and the domain data from the repository 115 of the EPSS. The domain data including the domain name, IP address or other domain related data is provided/collected by the server of host company, therefore, the domain data is considered as the data based on the transformation parameters (name, IP address or other related domain data…). Each of the subset of the small model is trained with different dataset, whose results are combined as the input to the master classifier, which is considered as the pre-defined framework of computer- executable instructions. The domain information is encoded for input into the machine learning model, that is corresponding to the model template code..).;
and tuning parameters of the first machine-learning model utilizing the training-tuning model template code base from the training dataset based on the first set of hyperparameters (Mcnee, [Par. 0030-0040], “” And “[0030], In FIG. 2, the pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application in order to put the EPSS build framework into context. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.). According to the pipeline 200, labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain (e.g., determined, selected, designated, etc.) machine learning algorithm 204 (such as linear regression) are input into a build process 210 to build a trained model instance 211 (a binary) . This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221. The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline (rebuilding the model instance 210 and executing the trained and tuned model 220) until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable (the validation data is used to validate the prediction of the test data as malicious or not)…0040, “In block 404, feature engineering is performed to engineer feature vectors for use in machine learning models. In overview, a feature vector is used as a way to select (extract, determine, or the like) and encode domain information for input into a machine learning algorithm. For example, a feature vector applied to a type of domain related data stored in the EPSS domain table (such as “domain name”), can be used to encode information about each domain based upon whether the domain matches or doesn't match specific criteria. One such feature vector may indicate whether a domain record has a domain name that includes a number or does not, or the ratio of letters to numbers in the domain name, or other such characteristics.”),” Examiner’s note, the parameter is tuning by using the training dataset a long with the model tuning parameters (training tuning model container) based on the hyperparameter (correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements). However, the claim does not define what is the model template code, therefore, based on the Broadest Reasonable Interpretational, the domain information is encoded for input into the machine learning model, that is equated as the model template code.).
However, McNee does not teach the first set of hyperparameter provided via the one or more client devices, receive, from one or more client devices, at least one request to train a first machine- learning model the at least one request comprising a first data identifier, a first set of transformation parameters and a first set of hyperparameters based on the at least one request, train the first machine-learning model utilizing a pre- defined framework of computer-executable instructions having a plurality of template code bases comprising a data generation model template code base, a data transformation model template code base receive, from at least one client device, at least one additional request to train a second machine-learning model the at least one additional request comprising a second data identifier, a second set of transformation parameters, and a second set of hyperparameters; and based on the at least one additional request, train the second machine-learning model utilizing the pre-defined framework of computer-executable instructions and the second data identifier, the second set of transformation parameters, and the second set of hyperparameters.
On the other hand, Yu teaches Hyperparameter provided via the one or more client devices (Yu, [par.0068], “The server 620 provides the hyperparameter tuning system 120 over the web to clients 605 through a network 640. By way of example, the client computing device is implemented and embodied in a personal computer 605a, a tablet computing device 605b or a mobile computing device 605c (e.g., a smart phone), or other computing device. Any of these examples of the client computing device are operable to obtain content from the store 616.”.
McNee and Yu are analogous in arts because they have the same field of endeavor of generating the machine learning model training and tuning system.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to have modify the and tuning parameters of the first machine-learning model utilizing the training-tuning model template code base from the training dataset based on the first set of hyperparameters , as taught by McNee, to include the first set of hyperparameter provided via the one or more client devices, as taught by Yu. The modification would have been obvious because one of the ordinary skills in art would be motivated to improve the speed of the machine learning model tuning system, (YU, [Par.0006], “Systems and methods are provided herein to enable improvements in speed and reductions in the computing resources expended when tuning hyperparameters. As described herein, knowledge from previous evaluations of candidate value sets for the hyperparameters are incorporated into the analysis and growth of the candidate value sets while directing the expenditure of computing resources to those candidate value sets that are most promising.”).
However, neither McNee nor Yu teaches receive, from one or more client devices, at least one request to train a first machine- learning model the at least one request comprising a first data identifier, a first set of transformation parameters and a first set of hyperparameters based on the at least one request, train the first machine-learning model utilizing a pre- defined framework of computer-executable instructions having a plurality of template code bases comprising a data generation model template code base, a data transformation model template code base receive, from at least one client device, at least one additional request to train a second machine-learning model the at least one additional request comprising a second data identifier, a second set of transformation parameters, and a second set of hyperparameters; and based on the at least one additional request, train the second machine-learning model utilizing the pre-defined framework of computer-executable instructions and the second data identifier, the second set of transformation parameters, and the second set of hyperparameters
On the other hand, Bendre teaches receive, from one or more client devices, at least one request to train a first machine- learning model (Bendre, [Par.0119], “By way of example, the scheduler device 604 could receive a first ML training request from a first computing system that enables use of a first customer instance as well as a second ML training request from a second computing system that enables use of a second customer instance. Responsively, the scheduler device 604 may assign the first ML training request to a first ML trainer process, which may cause a first ML trainer device to execute the first ML trainer process serving the first ML training request”)
the at least one request comprising a first data identifier, a first set of transformation parameters (Bendre, [Par.0125], “The scheduling controller 618 may be configured to initiate operations within the scheduler device 604 in response to receiving an ML training request. For example, the scheduling controller 618 may store information related to a received ML training request, such as an identifier of the customer instance from which the ML training request has been received and/or an identifier of a solution definition that provides basis for the ML training request, among others.” and [Par.0173-0174], “By way of example, the computing system 602 could receive a solution definition indicating training data and a target variable in line with the discussion above. In this example, the training data could be a plurality of data points each indicating an extent of remaining disk space at a respective point in time…Specifically, Table 1 shows training data corresponding to data points that indicate extent of remaining disk space respectively at each of various months of a given year. As shown, the remaining disk space is represented by the variable Y and the month is represented by the variable X. For instance, Table 1 shows that the enterprise network has 2.5 TB of remaining disk space in January (i.e., 1.sup.st month of the year), that the enterprise network has 2.1 TB of remaining disk space in February (i.e., 2.sup.nd month of the year), that the enterprise network has 2.0 TB of remaining disk space in March (i.e., 3.sup.rd month of the year), and so on.” Examiner’s note, t Examiner’s note, the first machine learning model request associates with the identifier of the customer instance and/or an identifier of a solution definition, that are corresponding to the first data identifier and the first set of transformation parameters.),
and a first set of hyperparameters (Bendre, [Par.0154], “In some implementations, the scheduler device 604 could receive recommendation(s) or may otherwise determine recommended ML trainer device(s) to which the scheduler device 604 could assign the ML training request 632. For instance, once the scheduler device 604 determines performance metric(s), the scheduler device 604 could determine a performance score respectively for each of a plurality of ML trainer devices. To do so for a given ML trainer device, the scheduler device 604 could assign a weight respectively to each performance metric determined for that given ML trainer device” Examiner’s note, each of the request associated with the particular weight for the given machine learning model, therefore, the wight associates with the first machine learning model request is considered as the first set of hyperparameters.);
based on the at least one request, train the first machine-learning model utilizing a pre- defined framework of computer-executable instructions having a plurality of template code bases comprising a data generation model template code base, a data transformation model template code base (Bendre, [Par.0119-Par.0131], “By way of example, the scheduler device 604 could receive a first ML training request from a first computing system that enables use of a first customer instance as well as a second ML training request from a second computing system that enables use of a second customer instance. Responsively, the scheduler device 604 may assign the first ML training request to a first ML trainer process, which may cause a first ML trainer device to execute the first ML trainer process serving the first ML training request… For instance, the scheduler device 604 may be configure to determine that the particular ML trainer process is available after completing serving of the first ML training request, and may then responsively assign the second ML training request to the particular ML trainer process. Other examples and cases are also possible… The ML trainer 626 process may take the form of any ML algorithm, code, routine or the like that is executable by the ML trainer device 606 to learn from training data, so as to create an ML model by a training process. Examples of ML trainer processes may include (without limitation): Decision Trees, Naïve Bayes Classification, Least Squares Regression, and Logistic Regression, among others. As such, the ML trainer 626 process may be any currently available and/or future developed ML trainer process arranged for the purpose of generating various types of ML models.” Examiner’ snote, the scheduler schedules the order of the executing of the machine learning model based on the other of the request, for example, the first machine learning model executing request is performed before the second machine learning model executing request, therefore, the scheduling to execute the machine learning model corresponding to the pre- defined framework of computer-executable instructions. The trainer device may execute the training algorithm or code to perform based on the given request.
receive, from at least one client device, at least one additional request to train a second machine-learning model (Bendre, [Par.0119], “By way of example, the scheduler device 604 could receive a first ML training request from a first computing system that enables use of a first customer instance as well as a second ML training request from a second computing system that enables use of a second customer instance. Responsively, the scheduler device 604 may assign the first ML training request to a first ML trainer process, which may cause a first ML trainer device to execute the first ML trainer process serving the first ML training request and may assign the second ML training request to a second ML trainer process, which may cause a second ML trainer device to execute the second ML trainer process serving the second ML training request.”),
the at least one additional request comprising a second data identifier, a second set of transformation parameters, and a second set of hyperparameters;
(Bendre, [Par.0125], “The scheduling controller 618 may be configured to initiate operations within the scheduler device 604 in response to receiving an ML training request. For example, the scheduling controller 618 may store information related to a received ML training request, such as an identifier of the customer instance from which the ML training request has been received and/or an identifier of a solution definition that provides basis for the ML training request, among others.” and [Par.0173-0174], “By way of example, the computing system 602 could receive a solution definition indicating training data and a target variable in line with the discussion above. In this example, the training data could be a plurality of data points each indicating an extent of remaining disk space at a respective point in time…Specifically, Table 1 shows training data corresponding to data points that indicate extent of remaining disk space respectively at each of various months of a given year. As shown, the remaining disk space is represented by the variable Y and the month is represented by the variable X. For instance, Table 1 shows that the enterprise network has 2.5 TB of remaining disk space in January (i.e., 1.sup.st month of the year), that the enterprise network has 2.1 TB of remaining disk space in February (i.e., 2.sup.nd month of the year), that the enterprise network has 2.0 TB of remaining disk space in March (i.e., 3.sup.rd month of the year), and so on.” Examiner’s note, the second machine learning model request associates with the identifier of the customer instance and/or an identifier of a solution definition, that is corresponding to the second data identifier and a second set of transformation parameters.),
and a second set of hyperparameters (Bendre, [Par.0154], “In some implementations, the scheduler device 604 could receive recommendation(s) or may otherwise determine recommended ML trainer device(s) to which the scheduler device 604 could assign the ML training request 632. For instance, once the scheduler device 604 determines performance metric(s), the scheduler device 604 could determine a performance score respectively for each of a plurality of ML trainer devices. To do so for a given ML trainer device, the scheduler device 604 could assign a weight respectively to each performance metric determined for that given ML trainer device” Examiner’s note, each of the request associated with the particular weight for the given machine learning model request, therefore, the weight is associated with the second machine learning request is considered as the second set of hyperparameters..);
and based on the at least one additional request, train the second machine-learning model utilizing the pre-defined framework of computer-executable instructions and the second data identifier, the second set of transformation parameters, and the second set of hyperparameters (Bendre, [Par.0119-Par.0131], “By way of example, the scheduler device 604 could receive a first ML training request from a first computing system that enables use of a first customer instance as well as a second ML training request from a second computing system that enables use of a second customer instance. Responsively, the scheduler device 604 may assign the first ML training request to a first ML trainer process, which may cause a first ML trainer device to execute the first ML trainer process serving the first ML training request, and may assign the second ML training request to a second ML trainer process, which may cause a second ML trainer device to execute the second ML trainer process serving the second ML training request.… For instance, the scheduler device 604 may be configure to determine that the particular ML trainer process is available after completing serving of the first ML training request, and may then responsively assign the second ML training request to the particular ML trainer process. Other examples and cases are also possible… The ML trainer 626 process may take the form of any ML algorithm, code, routine or the like that is executable by the ML trainer device 606 to learn from training data, so as to create an ML model by a training process. Examples of ML trainer processes may include (without limitation): Decision Trees, Naïve Bayes Classification, Least Squares Regression, and Logistic Regression, among others. As such, the ML trainer 626 process may be any currently available and/or future developed ML trainer process arranged for the purpose of generating various types of ML models.” Examiner’s note, the scheduler schedules the order of the executing of the machine learning model based on the other of the request, for example, the second machine learning model executing request is performed after the first machine learning model executing request, therefore, the scheduling to execute the machine learning model corresponding to the pre- defined framework of computer-executable instructions. The second machine learning model request associates with the identifier of the customer instance and/or an identifier of a solution definition, and particular assigned weight, that are corresponding to the second data identifier and a second set of transformation parameters and second set of hyperparameters, as it can be seen at (Bendre, [Par.0125], “The scheduling controller 618 may be configured to initiate operations within the scheduler device 604 in response to receiving an ML training request. For example, the scheduling controller 618 may store information related to a received ML training request, such as an identifier of the customer instance from which the ML training request has been received and/or an identifier of a solution definition that provides basis for the ML training request, among others.” And [Par.0154], “In some implementations, the scheduler device 604 could receive recommendation(s) or may otherwise determine recommended ML trainer device(s) to which the scheduler device 604 could assign the ML training request 632. For instance, once the scheduler device 604 determines performance metric(s), the scheduler device 604 could determine a performance score respectively for each of a plurality of ML trainer devices. To do so for a given ML trainer device, the scheduler device 604 could assign a weight respectively to each performance metric determined for that given ML trainer device” )
McNee, Yu and Bendre are analogous in arts because they have the same field of endeavor of generating the machine learning model training and tuning system.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to have modify theA non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause a computing device to, as taught by McNee, to include the receive, from one or more client devices, at least one request to train a first machine- learning model the at least one request comprising a first data identifier, a first set of transformation parameters and a first set of hyperparameters based on the at least one request, train the first machine-learning model utilizing a pre- defined framework of computer-executable instructions having a plurality of template code bases comprising a data generation model template code base, a data transformation model template code base receive, from at least one client device, at least one additional request to train a second machine-learning model the at least one additional request comprising a second data identifier, a second set of transformation parameters, and a second set of hyperparameters; and based on the at least one additional request, train the second machine-learning model utilizing the pre-defined framework of computer-executable instructions and the second data identifier, the second set of transformation parameters, and the second set of hyperparameters, as taught by Bendre. The modification would have been obvious because one of the ordinary skills in art would be motivated to improve the computer resource , (Bendre, [Par.0004], “Disclosed herein is a cloud-based network system that provides a remote ML arrangement, which can be shared among various enterprise networks. The remote ML arrangement can securely generate ML model(s) and prediction(s) that are based on given enterprise's data and are accessible only to client devices on the given enterprise's network. In this way, the network system could help an enterprise to save time, to improve use of computing resources, and/or to reduce costs on specialized software, among other possible outcomes.”).
Regarding claim 2, Mcnee teaches the non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to extract the set of data utilizing the data generation model template code by executing queries to retrieve digital information from a third-party software platform (Mcnee [Par.0037], [0090], “In block 401, the system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information (e.g., administrator and ownership information), IP (internet protocol) addresses, DNS record data, passive DNS activity data, scraped HTML content, TLS certificate information, and/or other domain related data. This data may be collected from a variety of sources and at different cadences and may be generated by the EPSS itself. For example, blocklist data which indicates known malicious domains, are available from a variety of services which update typically at least daily. For example, such data is available from organizations or companies such as The Spamhaus Project, an international organization that delivers lists of blocked IP addresses and blocked domains as soon as they are researched and added to their threat lists. Other private companies and other organizations provide similar data or subsets of such data. Other types of IID may be updated once a day or less frequently, for example some are streamed in near real-time, others are forwarded weekly, bi-weekly, monthly, etc. For example, public DNS (“A” record) data are available to all DNS servers participating in internet traffic, as they are the “directory” entries that map top-level logical names (such as “domaintools.com”) to IP addresses. Passive DNS activity data are packets that indicate that at some point in time a domain has been associated with a specific DNS record. This data is collected and distributed, for example, by a service such as a hosting company or other Internet Service Provider (ISP), which inserts a “probe” to detect such packets. Businesses that host their own DNS servers also can insert such probes and collect data similarly. Whois data is maintained by a distributed registry that stores information from ISP and other hosting services when a domain is registered. This data is typically obtained by using third party aggregator services that accumulate registration data from various registrars according to ICANN (“icann.org”) agreements, a non-profit organization responsible for administering IP addresses. For example, whois data may comprise attributes such as domain name, domain status, updated date, creation date, expiration date, registrar name/ID, registrant data (e.g., name, organization, address info, contact info, etc.), DNS security extensions indicator, and/or other information. Other information such as BGP (Border Gateway Protocol information), SSH keys, blockchain information, and the like may also be obtained and used to characterize domain data. Other IID may be made accessible or distributed similarly or in other manners, for example, by scraping HTML data from web pages using known or proprietary HTML content (web page) scraping tools, by accessing TLS certificate information, etc.” and [Par.0040-0046], “[0040], In block 404, feature engineering is performed to engineer feature vectors for use in machine learning models. In overview, a feature vector is used as a way to select (extract, determine, or the like) and encode domain information for input into a machine learning algorithm. For example, a feature vector applied to a type of domain related data stored in the EPSS domain table (such as “domain name”), can be used to encode information about each domain based upon whether the domain matches or doesn't match specific criteria. One such feature vector may indicate whether a domain record has a domain name that includes a number or does not, or the ratio of letters to numbers in the domain name, or other such characteristics. ... [0044], Domain table 500 comprises one or more domain records, represented by rows 501a-501g, one for each domain whose information has been collected. Columns 501-108 represent a type or category of internet infrastructure data (IID) such as domain name 501; IP address 502; DNS record data or other (geographic) zone data 503; status information such as whether the domain is known malicious (KM) or known neutral (KN) 505; values computed by EPSS predictive threat profilers as to whether the domain is predictably malicious or its classification or “risk” score(s) 506; TLS certificate information 507; whois data 508; hostname data 509; passive DNS (pDNS) activity 510, or any other type of IID. Notably, as described above, each of these columns may represent one or more other columns/values. ..TLS data 507 comprises multiple fields/columns such as the name of the issuing certificate authority, alternate domain names, issue date, expiry data, public key, digital signature of the certificate authority, etc… [0046], FIG. 6 is a block diagram of data abstractions used by an example Enhanced Predictive Security System for feature engineering…In the abstraction hierarchy shown, feature classes 601 are used to query the IID for specific data regardless of how the answer (extracted data) is encoded, feature class vectors 602 encode the extracted data according to a specific algorithm, and feature vectors 603 aggregate (concatenate, combine, collect, etc.) the feature class vectors 602 of relevance for a particular purpose into a single vector called a feature vector 603.” Examiner’s note, domain data is extracted based on the metadata of the EPSS by query the IID to retrieve the digital information (the information on the domain table) from the using third party aggregator services that accumulate registration data from various registrars according to ICANN (“icann.org”) agreements, a non-profit organization responsible for administering IP addresses. The domain information is encoded for input into the machine learning model that is corresponding to the model template code.).
Regarding claim 3, McNee teaches the non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the training dataset by executing computer-executable instructions within the data transformation model template code base to convert the set of data into a predetermined data format according to the first set of transformation parameters (Mcnee, [Par.0046, 0090], “FIG. 6 is a block diagram of data abstractions used by an example Enhanced Predictive Security System for feature engineering. In one example EPSS configuration, data abstraction hierarchy 600 includes a three level architecture for each IID, which comprises one or more feature classes 601a-601b, one or more feature class vectors 602a-602b, and one or more feature vectors 603. In the abstraction hierarchy shown, feature classes 601 are used to query the IID for specific data regardless of how the answer (extracted data) is encoded, feature class vectors 602 encode the extracted data according to a specific algorithm, and feature vectors 603 aggregate (concatenate, combine, collect, etc.) the feature class vectors 602 of relevance for a particular purpose into a single vector called a feature vector 603. A feature vector, e.g., feature vector 603, is what is fed into a machine learning algorithm as input. In one EPSS configuration, no more than one encoding of a particular feature class (FCV) is included in a resultant feature vector for a given ML algorithm instance and all of the feature class vectors are concatenated to derive the resultant feature vector. (In this example there may be derived feature classes that include different encodings of a feature class also included by itself in the resultant feature vector.) Other EPSS configurations may combine feature class vectors into feature vectors differently.” Examiner’s note, converting/encoding the domain data into the specific feature vector based on the machine learning algorithm, and the defined feature vector is used to input into the machine learning model, therefore, the specific feature vector is defined based on the machine learning algorithm is considered as the predetermined format.)
Regarding claim 4, McNee teaches the non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to tune the parameters of the machine-learning model by utilizing the training-tuning model template code to (Mcnee, [Par.0020, 0030,0040, 0090], “[0030], In FIG. 2, the pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application in order to put the EPSS build framework into context. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.). According to the pipeline 200, labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain (e.g., determined, selected, designated, etc.) machine learning algorithm 204 (such as linear regression) are input into a build process 210 to build a trained model instance 211 (a binary) . This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221. The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline (rebuilding the model instance 210 and executing the trained and tuned model 220) until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable (the validation data is used to validate the prediction of the test data as malicious or not).” Examiner’s note, the parameter is tuning by using the training dataset a long with the model tuning parameters (training tuning model container).).
generate, for the training dataset, training predictions utilizing the hyperparameters (McNee, [Par.0030, 0090], In FIG. 2, the pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application in order to put the EPSS build framework into context. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.). According to the pipeline 200, labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain (e.g., determined, selected, designated, etc.) machine learning algorithm 204 (such as linear regression) are input into a build process 210 to build a trained model instance 211 (a binary) . This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221. The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline (rebuilding the model instance 210 and executing the trained and tuned model 220) until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable (the validation data is used to validate the prediction of the test data as malicious or not).” Examiner’s note, the training prediction is generated based on the hyperparameter, for example, the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable/hyperparameter” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.).)
and modify one or more learned parameters of the first machine-learning model based on a comparison between ground truth data and the training predictions (Mcnee, [Par.0030, 0090], “In FIG. 2, the pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application in order to put the EPSS build framework into context. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.). According to the pipeline 200, labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain (e.g., determined, selected, designated, etc.) machine learning algorithm 204 (such as linear regression) are input into a build process 210 to build a trained model instance 211 (a binary) . This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221. The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline (rebuilding the model instance 210 and executing the trained and tuned model 220) until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable (the validation data is used to validate the prediction of the test data as malicious or not). The data used as training, test, or validation data can be sampled as described according to FIG. 3. This loop continues until a prediction/result 221 is generated that is considered within acceptable characteristics as described above. When an acceptable trained model state is achieved, trained model instance 211 can be deployed in an application (model execution 220) with new (unlabeled data) domain data 214 to generate a prediction/result 230. This prediction/result 230 can then be forwarded and/or used in any appropriate manner such as to inform end users of a predictably malicious domain, to rank domains as malicious, or the like.” Examiner’s note, the parameter is tunning until the prediction result is considered as the acceptable based on the prediction result and validation data.).
However, McNee does not teach the first set of hyperparameters provided via the one or more client devices,
On the other hand, Yu teaches the first of hyperparameters provided via the one or more client devices (Yu, [par.0068], “The server 620 provides the hyperparameter tuning system 120 over the web to clients 605 through a network 640. By way of example, the client computing device is implemented and embodied in a personal computer 605a, a tablet computing device 605b or a mobile computing device 605c (e.g., a smart phone), or other computing device. Any of these examples of the client computing device are operable to obtain content from the store 616.”);
McNee and Yu are analogous in arts because they have the same field of endeavor of generating the machine learning model training and tuning system.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to have modify the generate, for the training dataset, training predictions utilizing the hyperparameters, as taught by McNee, to include the hyperparameters provided via the one or more client devices, as taught by Yu. The modification would have been obvious because one of the ordinary skills in art would be motivated to improve the speed of the machine learning model tuning system, (YU, [Par.0006], “Systems and methods are provided herein to enable improvements in speed and reductions in the computing resources expended when tuning hyperparameters. As described herein, knowledge from previous evaluations of candidate value sets for the hyperparameters are incorporated into the analysis and growth of the candidate value sets while directing the expenditure of computing resources to those candidate value sets that are most promising.”).
Regarding claim 6, McNee teaches the non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to register, utilizing a registration template code based, a trained version of the first machine-learning model with a model storage memory (Mcnee, [Par.0054], “For example, in order to instantiate the model for Application(j), the following activities are performed. First, the appropriate model subsets are either designed and built according to process 801 or selected and instantiated from the model library 880. The process for building new model subsets is described further with respect to FIG. 9. In essence, in order to build and train a new model subset such as subset 810, feature classes are selected from a feature class library and applied to sampled domain data which are then transformed into feature vectors. The feature vector for each of the “i” models 811i in model subset 810 has the same fields (what values of the IIDS are being looked at and encoded) but the actual values that correspond to the sampled training data may differ as these values are data dependent. In addition, each model of the models 811i in subset 810 (for example, model 811n) uses the same machine learning algorithm (such as linear or logistic regression, SVMs, naïve bayes, Bayesian belief networks, decision tress, random forests, neural networks, and the like) and the same hyper parameters for tuning the indicated machine learning algorithm, but uses different training data (separate samples). The model can be built according to pipeline 200 described with reference to FIGS. 2 and 3. As well, the sampling of the data can be performed using the improved neutral data sets as described with reference to FIGS. 10 and 11. Once the subset is built and trained it can be stored in model library 880. Each model subset stored in the library 880 has metadata stored with it so that they model subset can easily be instantiated as needed for other applications. Stored model metadata 881 includes for example, for each new subset model, an indication of a machine learning algorithm, a set of hyper parameters for tuning the indicated machine learning algorithm, a description of feature class information used to build an associated input feature vector, an indication of a source for training data, and an indication of training data sampling parameters and any other metadata needed to recreated the model.” Examiner’s note, the model is trained and stored in the model library. Each of subset of model has the indications of machine learning algorithm, a set of hyper parameters for tuning the indicated machine learning algorithm, a description of feature class information used to build an associated input feature vector, therefore, each model subset is considered as the registration template code base.).
Regrading claim 7, McNee teaches the non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate, utilizing a statistics generation model template code based of the pre-defined framework of computer executable instruction, feature statistics for values of the training dataset (McNee, [Par.0053], The process for building new model subsets is described further with respect to FIG. 9. In essence, in order to build and train a new model subset such as subset 810, feature classes are selected from a feature class library and applied to sampled domain data which are then transformed into feature vectors. The feature vector for each of the “i” models 811i in model subset 810 has the same fields (what values of the IIDS are being looked at and encoded) but the actual values that correspond to the sampled training data may differ as these values are data dependent. In addition, each model of the models 811i in subset 810 (for example, model 811n) uses the same machine learning algorithm (such as linear or logistic regression, SVMs, naïve bayes, Bayesian belief networks, decision tress, random forests, neural networks, and the like) and the same hyper parameters for tuning the indicated machine learning algorithm, but uses different training data (separate samples).” Examiner’ s note, feature classes are selected from the feature class library and transformed into the feature vector to apply to the training data, therefore, the feature class is considered as the feature statistic.).
Regrading claim 8, McNee teaches the non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to utilize a pre-defined inference framework of computer-executable instruction comprising a feature generation model template code base, a feature transformation model template code base, and an inference generation model template code base to generate a first prediction from the first machine-learning model (McNee, [Par.0050-0053, Fig.8, 0090], “FIG. 8 is a block diagram illustrating an improved architecture for building, training, and running an example Enhanced Predictive Security System… [0052] For example, as EPSS architecture 800 illustrates how models for three different applications, Application(j), Application(k), and Application(y) can be built (e.g., developed and instantiated) and deployed. These applications may correspond for example to an application for phishing, spam, or malware, or may comprise the same type of application (e.g., phishing) for different target customer or the like. Each of the ensemble classifiers for these applications, for example classifiers 812, 821, and 831, may be built and deployed using model library 880 and may employ a single level ensemble master classifier (such as classifier 821 for Application(k) and classifier 831 for Application(y)) or may employ a multi-level ensemble master classifier such as for Application(j).” Examiner’s note, each of the application is considered as the predefined inference model workflow comprising input vector 850 (feature generation model template code), the ensemble classification engine 870 (a feature transformation model template code), the classifier 812, 821 or 831 ( inference generation model template code), to generate the machine learning model.) .
Regarding claim 9, McNee teaches the non-transitory computer-readable medium of claim 8, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the first prediction from the first machine-learning model utilizing the pre-defined inference framework of computer executable instructions by (McNee, [Par.0090], “Some or all of the components and/or data structures may be stored on tangible, non-transitory storage mediums. Some or all of the system components and data structures may also be stored as data signals (e.g., by being encoded as part of a carrier wave or included as part of an analog or digital propagated signal) on a variety of computer-readable transmission mediums, which are then transmitted, including across wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of this disclosure may be practiced with other computer system configurations.”):
extracting inference data features utilizing the feature generation model template code base of the pre-defined inference framework of computer executable instructions utilizing feature identifiers provided via the one or more client devices (McNee, [Par.0062], “FIG. 9 is a block diagram detailing the process for feature class selection and feature vector transformation used by the example Enhanced Predictive Security System. As described above, FIG. 9 illustrates further detail on process 801 for determining feature vectors to be used with the model subsets (the weak classifiers). Accordingly, a set of feature classes is selected 905 either using feature engineering as described with reference to block 404 in FIG. 4, or from a feature class library 903 which contains definitions (and optionally metadata) resulting from such feature engineering—stored for easy reference and access. The selected feature classes are then applied against sampled data (logic 901 and 902) to obtain appropriate values for the sampled data. This data is then (encoded and) transformed into a feature vector 909 for use with a model subset.”) ;
generating encoded features from the inference data features utilizing the feature transformation model template code base of the pre-defined inference framework of computer executable instruction (McNee, [Par.0052-0056], “The feature vector for each of the “i” models 811i in model subset 810 has the same fields (what values of the IIDS are being looked at and encoded) but the actual values that correspond to the sampled training data may differ as these values are data dependent….[0053], For example, in order to instantiate the model for Application(j), the following activities are performed. First, the appropriate model subsets are either designed and built according to process 801 or selected and instantiated from the model library 880. The process for building new model subsets is described further with respect to FIG. 9. In essence, in order to build and train a new model subset such as subset 810, feature classes are selected from a feature class library and applied to sampled domain data which are then transformed into feature vectors. The feature vector for each of the “i” models 811i in model subset 810 has the same fields (what values of the IIDS are being looked at and encoded) but the actual values that correspond to the sampled training data may differ as these values are data dependent…For example, for Application(j), model subset output 813, 822, and 832 are configured as input 850 to the ensemble master classifier 812. In the example shown, the results of each model subset output 813, 822, and 832 are input into input vector 850. Each of these results is then initially weighted by some amount specified in weight vector 860 before being input into the ensemble classification engine 870. The ensemble classification engine 870 may be, for example, a deep neural network or other machine learning algorithm. These initial weights may be formulated using a variety of rules including initially weighting them all the same (flat weighting), weighting the inputs according to their contributions to the input vector 850 or their inverse contributions to the input vector 850, weighting them for example according to the Sn support or confidence values, or some combination of any of the above.” Examiner’s note, using the input vector to input the feature vector is encoded by the feature class is selected from the feature class library.
and generating the first prediction for the encoded features utilizing the inference generation model template code base of the pre-defined inference framework of computer executable instructions (McNee, [Par.0053-0056], “The model subset output from each of the model subsets is then configured to be fed into an ensemble master classifier for that particular application so that the predictions can be reduced to a single (final) score. For example, for Application(j), model subset output 813, 822, and 832 are configured as input 850 to the ensemble master classifier 812. In the example shown, the results of each model subset output 813, 822, and 832 are input in to input vector 850. Each of these results is then initially weighted by some amount specified in weight vector 860 before being input into the ensemble classification engine 870. The ensemble classification engine 870 may be, for example, a deep neural network or other machine learning algorithm. These initial weights may be formulated using a variety of rules including initially weighting them all the same (flat weighting), weighting the inputs according to their contributions to the input vector 850 or their inverse contributions to the input vector 850, weighting them for example according to the Sn support or confidence values, or some combination of any of the above. Other weighting values may be incorporated. For example, if 6 individual models contribute to model subset 810, 3 contribute to model subset 820, and 5 contribute to model subset 830, then any one model contributes on 1/14 into input vector 850 (if all equal) and the weights chosen according (0.07 each), or each result of model subset 810 could be viewed as contributing ⅙ (0.17) to the input, subset 820 contributing ⅓ (0.33) to the input, and subset 830 contributing ⅕ (0.2) to the input (inverse weighting). Alternatively, based upon the importance of a particular model subset, the weightings (even if Sn values are used) may be skewed as desired. Other weighting combinations appropriate to the application can be similarly incorporated. For example, an initial logistic regression or an initial iteration of classifier 870 may be run on input 850 and the coefficients used as weights vector 860.” Examiner’s note, the encoded feature vector as the input into the classifier (inference generation model container) to generate the prediction for the application (pre-defined inference model container workflow).
and the parameters of the first machine-learning model tuned utilizing the training-tuning model template code base (Mcnee, [Par.0020, 0030. 0040], “[0020] Each subset of the smaller models includes multiple instances of a same model sharing a same machine learning algorithm, modeling tuning parameters, and feature vector values but trained using different trained data. Hence each model subset acts as a set of “weak classifiers” for a particular type or collection of threat data. Certain subsets may be more applicable and tuned for certain types of applications because they pull (access, assess, determine, etc.) different domain related data, or are tuned differently, etc. A combination of the results of each applicable subset of weak classifiers then is fed as input into the ensemble master classifier, which can be iteratively run with varying weights applied to the weak classifier subset outputs until a determined optimization value (e.g., a threshold, minimum, percentage, probability, etc.) is reached.” And “[0030], In FIG. 2, the pipeline 200 illustrates how models are built and tuned for deployment as a cybersecurity threat analysis application in order to put the EPSS build framework into context. Portions of the pipeline 200 are looped and assessed (or reassessed) until the executed model 220 is capable of predicting a result 221 that is considered “acceptable” (e.g., correct according to some determined value, percentage of time, threshold, precision and/or recall statistical requirements, etc.). According to the pipeline 200, labeled (known) malicious data 201 along with labeled (known) neutral data 202 in the form of training data 203 along with model tuning parameters 205 and a certain (e.g., determined, selected, designated, etc.) machine learning algorithm 204 (such as linear regression) are input into a build process 210 to build a trained model instance 211 (a binary) . This trained model instance 211 (i.e., trained model) is then run (shown as model execution 220) on labeled malicious and neutral test data 212 to generate a prediction/result 221. The resultant prediction 221 is input along with labeled malicious and neutral validation data 216 into a tuning system 215, which is used to determine the (potentially modified) model tuning parameters 205 to run in the next iteration of the pipeline (rebuilding the model instance 210 and executing the trained and tuned model 220) until the trained model 220 predicts an outcome (result) that is correct sufficient times and with sufficient accuracy to be considered acceptable (the validation data is used to validate the prediction of the test data as malicious or not).” Examiner’s note, the parameter is tuning by using the training dataset a long with the model tuning parameters (model template code base)).
Regarding claim 10 is rejected for the same reason as the claim 1, since these claims recite the same limitations.
Regarding claim 11 is rejected for the same reason as the claim 2, since these claims recite the same limitations.
Regarding claim 12 is rejected for the same reason as the claim 3, since these claims recite the same limitations.
Regarding claim 13 is rejected for the same reason as the claim 4, since these claims recite the same limitations.
Regarding claim 15 is rejected for the same reason as the claim 6, since these claims recite the same limitations.
Regarding claim 16 is rejected for the same reason as the claim 7, since these claims recite the same limitations.
Regarding claim 17 is rejected for the same reason as the claim 1, since these claims recite the same limitations.
Regarding claim 18 is rejected for the same reason as the claim 2, since these claims recite the same limitations.
Regarding claim 19 is rejected for the same reason as the claim 3, since these claims recite the same limitations.
Regarding claim 20 is rejected for the same reason as the claim 4, since these claims recite the same limitations.
Claims 5, 14 are rejected under 35 U.S.C. 103 as being unpatentable over McNee et al. (Pub. No. US 20220150275-hereinafter, McNee) and further in view of YU et al. (PUB. No. US 20180121814 -hereinafter, Yu) and further in view of Bendre et al. (PUB. No. US. 20180322417-Hereinafter, Bendre) and further in view of Quanz et al. (PUB. No. US 20220147669 -hereinafter, Quanz).
Regarding claim 5, McNee teaches the non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to: generate a test dataset utilizing the data generation model template code base of the pre-defined framework of computer executable from the set of data based on the first set of transformation parameters provided via the one or more client devices (Mcnee, [Par.0024-0026, 0040, 0090], “FIG. 1 is a block diagram of components of an example Enhanced Predictive Security System described herein. In one example embodiment, the Enhanced Predictive Security System comprises one or more functional components/modules that work together to provide an architecture and a framework for building and deploying cybersecurity threat analysis application. For example, the EPSS 100 may comprise one or more machine learning algorithms 110, feature class engines (for use with feature engineering) 111, tuning systems 112, ensemble classifier engines 113, and validation and testing engines 114. These components cooperate and act upon domain data and feature class vectors (stored in a repository 115), to create sampled test, training, and validation data 116 and to build model subsets and applications using trained model library 117. In an example EPSS configuration, the trained model library 117 stores definitions of each model subset for easy re-instantiation, including an indication of the machine learning algorithm used to create the model along with hyper parameters for tuning the model, and a description of the feature class information used to build an input feature vector associated with the model, an indication of a source for training data, and an indication of training data sampling parameters. Other versions of the model library 117 may contain more or less or different information…” and [0037] In block 401, the system collects domain related data from Internet Infrastructure Data (IID), both gathered and derived, including for example, domain names, “whois” protocol information (e.g., administrator and ownership information), IP (internet protocol) addresses, DNS record data, passive DNS activity data, scraped HTML content, TLS certificate information, and/or other domain related data. This data may be collected from a variety of sources and at different cadences and may be generated by the EPSS itself. For example, blocklist data which indicates known malicious domains, are available from a variety of services which update typically at least daily. For example, such data is available from organizations or companies such as The Spamhaus Project, an international organization that delivers lists of blocked IP addresses and blocked domains as soon as they are researched and added to their threat lists…This data is collected and distributed, for example, by a service such as a hosting company or other Internet Service Provider (ISP)” Examiner’s note, the training data is created/generated based on feature class vector and the domain data from the repository 115 of the EPSS. The domain data including the domain name, IP address or other domain related data is provided/collected by the server of host company, therefore, the domain data is considered as the data based on the transformation parameters (name, IP address or other related domain data…), the repository is considered as the data generation model template code base of the system.);
However, McNee does not teach and determine, based on the test dataset, a machine-learning testing metric of the first machine- learning model utilizing an evaluation template code base of the pre-defined framework of computer executable instruction based on testing parameters provided via the one or more client devices,
On the other hand, Quanz teaches a and determine, based on the test dataset, a machine-learning testing metric of the first machine- learning model utilizing an evaluation template code base of the pre-defined framework of computer executable instruction based on testing parameters provided via the one or more client devices, (Quanz, [par.0031], “By virtue of the teachings herein, an entity can upload their data, specify set of models to try, time periods to train and evaluate, and efficiently receive forecasting model evaluation results and forecasting models to deploy and use.” , [Par.0042], “System 200 illustrates that there are three main actions performed by the efficiency engine 103 to automatically partitioning time series to run time series modeling in parallel. First, the time series data 202 is received by the efficiency engine 103 and a hierarchical partitioning is performed. The partitioning is hierarchical in that there may be partitions with larger group sizes (and fewer total groups) and sub-partitions with smaller group sizes (and more total groups). For example, a largest group size partition 207 may include a most loose criteria for inclusion (e.g., same region, same store, etc.,) and therefore include the largest group (i.e., time series (ts 1 to ts 10) in the present example). A tighter partition group represents a set of times series that is more likely to be related and benefit from cross-series modeling 209. The tightest partition group is referred to herein as a level 1 partition group (or grouping) and the level increases as the groups introduce more time series. The first level partition has tighter criteria for inclusion (e.g., same product line in a region) (e.g., 211, 213).” And [Par.0046], “In one embodiment, the efficiency engine 103 performs a test by performing partial modeling for a subset (e.g., one or more groups) from each level from the set of candidate levels (in parallel) to test accuracy and computation time for each. In this way, the computational capability is determined. Upon determining the computational capability of the computing device performing the processing of the time series data, a partitioning level is selected that can accommodate the processing of the time series data in a predetermined time period and a predetermined threshold accuracy. In the example of FIG. 2, the efficiency engine 103 determines that a level 2 partition (which includes group 1 (e.g., 215) and group 2 (e.g., 209) as 2 different groups in the partition to be modeled separately and simultaneously) provides the better accuracy and efficiency—and this can be based on simply testing a subset of groups at the level (e.g., level 2 being tested) initially—such as group 1 (e.g., 215) only, for a subset of modeling configurations, and comparing to similar tests at other levels. For example, the partitioning levels to choose from may be: level 0—which includes each individual time series in its own group—in which case a model is fit separately to each individual time series; level 1—which includes 211, 213, and 215 as 3 different groups—a separate model is fit to each of the 3 groups in parallel; level 2, which includes 209 and 215 as 2 different groups in that level that can be modeled independently; and level 3=207—which corresponds to fitting one model across all time series (modeling the whole set of time series together). Moving up or down this hierarchy results in different results in both terms of accuracy and efficiency—both might increase from level 0 up to a point (say level 1 or 2) and start to decrease for higher levels, for example.” Examiner’s note, determine based on the test data, the way to partial modeling for a subset (e.g., one or more groups) from each level (test metric), by the using evaluation model container (the time series data) of the system 200 , (Par.0042]. based on the predetermined time predetermined time period and a predetermined threshold accuracy (testing parameter), wherein, the predetermine time period provided via the user device (entity) ).
McNee, Yu and Quanz are analogous in arts because they have the same field of endeavor of generating the machine learning model.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to have modify the generate a test dataset utilizing the data generation model container of the pre-defined model container workflow from the set of data based on the transformation parameters provided via the one or more client devices, as taught by McNee, to include the determine, based on the test dataset, a machine-learning testing metric of the machine- learning model utilizing an evaluation model container of the pre-defined model container workflow based on testing parameters provided via the one or more client devices, as taught by Quanz. The modification would have been obvious because one of the ordinary skills in art would be motivated to improve the data forecasting (Quanz, [Par.0030], “In one aspect, the teachings herein make the forecasting for large numbers of time series and large data with state-of-the-art forecasting techniques both scalable and effective (i.e., computationally feasible on a given computing platform and improving accuracy thereof) by automatically determining an appropriate partition level of time series to perform cross-series modeling in parallel, where each partition forms a forecasting task that can be run in parallel. Additionally, the teachings herein facilitate cross-time-series machine learning algorithms to be leveraged, which provide the latest state-of-the-art approaches to forecasting, as well as modeling or sharing model parameters across time series and multi-task and/or multivariate models, to improve forecasting accuracy, as well as to including the growing amount of exogenous data for external factors like weather, events, social media, etc.”).
Regarding claim 14 is rejected for the same reason as the claim 5, as these claims recite the same limitations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure is provide below.
DACOSTA et al. (Pub. No.:us 20220138564-hereinafter, DACOSTA) teaches the machine learning pipeline .
NATARAJAN et al. (Pub. No.:us 20220114301-hereinafter, Fishel) teaches train the autonomous machines or clusters of autonomous machines for various predefined tasks with an intention to optimize AI/ML based task performance model with an offline training, then integrate pre-trained AI/ML based task performance modelsb.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EM N TRIEU whose telephone number is (571)272-5747. The examiner can normally be reached on Mon-Fri from 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached on (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.T./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights