DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 21-40 are pending and are examined herein.
Claims 21-40 are rejected under 35 USC 103.
Claims 21-40 are rejected on the grounds of Non-Statutory Double Patenting.
Information Disclosure Statement
The attached information disclosure statement(s) (IDS) is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the attached information disclosure statement(s) is/are being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not /be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 21-39 are rejected under 35 U.S.C. 103 as being unpatentable over “Gibiansky” (US 2016/0110657 A1) in view of “Chen” (US 2019/0362222 A1), further in view of “Choi” (Generating Multi-label Discrete Patient Records using Generative Adversarial Networks), and further in view of “Koch” (US 2018/0240041 A1).
Regarding claim 21, Gibiansky teaches
A scalable system for completing a model task using a serverless architecture, the system comprising: (Gibiansky, Abstract describes a system and method for performing machine learning. [0058] describes the system being operable on the cloud (i.e., a scalable, serverless architecture))
a model optimizer comprising: memory for storing instructions; and one or more processors configured to execute the stored instructions to perform operations comprising: (Gibiansky, [0010] describes an implementation as a system (understood to be a model optimizer) comprising memory storing instructions and processors.)
receiving a request to complete a model task; (Gibiansky, Fig. 4, step 402 shows a step of receiving data. Fig. 5 provides an example of the GUI used by the users1 to input this data to the system. This example is described at [0104-0105], and provides an example in which the task is to classify emails. In particular, the end of [0105] provides an example of a request input by a user.)
retrieving a first model from a model storage by selecting, ..., the first model... (Gibiansky, [0076] describes updating a partially or previously trained model. The “parameters” of Gibiansky are described at [0073-0078,0114] and include at least model type, “architectural hyperparameters” (e.g. [0075]: type of kernel and margin width; [0076]: number of trees), “training hyperparameters” (e.g., [0075]: whether or not to perform bagging), and values of model parameters (e.g. alpha coefficients in an SVM setting, see [0047]). As described at [0047], a model is identified with the parameter values which define it. [0078] describes retrieving this data from storage.)
retrieving a first hyperparameter based on the first model ...; (Gibiansky, [0076] describes updating a partially or previously trained model. The “parameters” of Gibiansky are described at [0073-0078,0114] and include at least model type, “architectural hyperparameters” (e.g. [0075]: type of kernel and margin width; [0076]: number of trees), “training hyperparameters” (e.g., [0075]: whether or not to perform bagging), and values of model parameters (e.g. alpha coefficients in an SVM setting, see [0047]). As described at [0047], a model is identified with the parameter values which define it. [0078] describes retrieving this data from storage. [0076] includes an example in which a model was trained on older data and is to be updated with newer data. See also Fig. 3, step 302 and [0098].)
provisioning computing resources to a first development instance configured to train the first model based on the first hyperparameter...; (Gibiansky, [0058] provides an example in which the servers are implemented in the cloud (i.e., a serverless architecture) as virtual machines. [0091] describes running optimization servers 102 in parallel to handle different models. Each of the optimization servers 102 is understood to be a development instance. [0093] indicates that this includes training the model based on retrieved data discussed earlier. See also Fig. 3, step 304 and [0098], where utilizing the parameters is understood to include training/updating the model.)
creating an instance of the first model from the first development instance; (Gibiansky, [0093] indicates that the machine learning method is implemented and that a model is trained. [0092] describes generating new parameter configurations. Since a model is specified by its parameters, it is understood that implanting a machine learning method according to a set of parameters is creating a model instance. As described at [0092], this is performed by the optimization server 102. See also Fig. 3, step 304 and [0098], where utilizing the parameters is understood to include training/updating the model.)
retrieving, by the first development instance, training data... (Gibiansky, [0046] describes receiving training data.)
training the instance of the first model by the first development instance using the training data to obtain a first trained model; (Gibiansky, [0092] describes “perform[ing] one or more iterations of forming a new parameter distribution, generating new parameters based on the new distribution and determining whether a stop condition is met”. [0046] indicates that the tuning/training is based on received training data.)
terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a dataset generated by the trained model and a reference dataset, a data similarity score between the dataset generated by the trained model and the reference dataset, or a data quality score for the dataset generated by the trained model; (Gibiansky, Fig. 3, step 310 and [0099]: the method of Fig. 3 (for training/fine tuning the model) is terminated when a stop condition is met. [0086-0089] describes various stop conditions which may be used. In particular, the termination criterion may be based on an accuracy of the model, which represents both (1) a similarity between the dataset generated (i.e., the predictions) and the reference dataset (i.e., the ground truth labels) and (2) a quality score for the predicted labels (with higher accuracy indicating higher quality).)
generating, by the first development instance, a first performance metric of the trained model; (Gibiansky, [0086-0087] describe one of the stopping criteria being based on a measure of fitness. [0091] describes the distributed scenario in which each optimization server manages data related to the model which it is optimizing. [0096] indicates that the coordination of the various optimization servers may be achieved by the data management unit 260.)
receiving, from the first development instance, the trained model and the first performance metric; (Gibiansky, [0086-0087] describe one of the stopping criteria being based on a measure of fitness. [0091] describes the distributed scenario in which each optimization server manages data related to the model which it is optimizing. [0096] indicates that the coordination of the various optimization servers may be achieved by the data management unit 260. It is understood that for the data management unit 260 to manage this data, it must be received from the optimization servers.)
receiving, from a second development instance, a second performance metric associated with a second model; (Gibiansky, [0086-0087] describe one of the stopping criteria being based on a measure of fitness. [0091] describes the distributed scenario in which each optimization server manages data related to the model which it is optimizing. [0096] indicates that the coordination of the various optimization servers may be achieved by the data management unit 260. It is understood that for the data management unit 260 to manage this data, it must be received from the optimization servers. Since there are a plurality of optimization servers working on different models in the example of [0091], each of these servers will generate data to send to the data management unit.)
determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and (Gibiansky, [0088] describes comparing the performance criteria of various models to stop poorly performing models. This is shown in Figure 3, step 310. The determination step 310 forms a loop with steps 306 and 308. The last time through this loop is understood to be the time which corresponds to this step.)
Gibiansky does not appear to explicitly teach
selecting, based on the model task, the first model in an index of stored models;
retrieving a first hyperparameter based on the first model and the model task;
...train the first model based on the first hyperparameter and the model task
...training data comprising synthetic data based on actual data;
...terminating the first development instance based on the determination that the termination condition is satisfied.
However, Chen—directed to analogous art--teaches
selecting, based on the model task, the first model in an index of stored models; retrieving a first hyperparameter based on the first model and the model task; ...train the first model based on the first hyperparameter and the model task (Chen, Fig. 8 shows a method for accessing (i.e., retrieving) a machine learning model based at least in part on schema data. This is described at [0069-0074]. In particular, step 812 shows a step of accessing a machine learning model usage from element 316 (and more generally from the Machine learning server 320). [0042] indicates that the model usage includes the model and associated hyperparameters and task. The machine learning server is further described at [0039-0042]. [0042] indicates that the model usage includes a description of task performed by the model, so this information is understood to be part of the index as described above. [0045] and Figure 3 and indicates that the models may be updated/retrained, which is based on the selecting described at [0042] which is based at least in part on the task.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which the invention pertains to modify the combination of Gibiansky to use the model storage method/system as taught by Chen because this allows for the time-intensive task of selecting a machine learning model by a person to be automated as described at [0002] and [0004] of Chen.
The combination of Gibiansky and Chen does not appear to explicitly teach
...training data comprising synthetic data based on actual data;
...terminating the first development instance based on the determination that the termination condition is satisfied.
However, Choi—directed to analogous art--teaches
...training data comprising synthetic data based on actual data; (Choi, Abstract describes using a Generative Adversarial Network (GAN) to generate synthetic data which may be used to perform predictive modeling tasks. Section 3 provides an overview of the method. Section 3.6., Presence disclosure, indicates that the synthetic data may be useful in preventing an attacker from determining that machine learning model was trained using a particular piece of data. Section 3.2. provides an overview of the GAN framework. In particular, the GAN is trained using actual data to generate synthetic data. That is, the synthetic data is generated based on the actual data. Appendix F discusses whether or not the data sets are dissimilar enough to fool an attacker. In particular, Figure 11 (and surrounding discussion) indicates that the attacker’s sensitivity is below 0.2 (with a Hamming distance of 0) and the precision is below 0.8 (with a Hamming distance of 0). Either of these indicates that a similarity between the synthetic data and actual data is less than a threshold (i.e., 0.2 or 0.8). Examiner notes that the claim requires that there be some similarity threshold for which the similarity between the synthetic data and actual data is less than that threshold, which is always true when the threshold is not predetermined.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which the invention pertains to modify the combination described above to generate synthetic data based on actual data as taught by Choi because this allows for large quantities of data to be made accessible while still respecting privacy concerns as described by Choi in the first two paragraphs of the introduction.
The combination of Gibiansky, Chen and Choi does not appear to explicitly teach
...terminating the first development instance based on the determination that the termination condition is satisfied.
However, Koch—directed to analogous art--teaches
...terminating the first development instance based on the determination that the termination condition is satisfied. (Koch, Fig. 6A-6C show a method for creating sessions. Each session trains a model which includes a parameter search. Step 674 is a cleanup step at the end. As described at [0195], this includes termination of sessions. This occurs in response to decision point 650, where it is determined whether or not the process should continue. In the combination with Gibiansky, Gibiansky is understood to teach the termination condition as described above and Koch is understood to teach the termination comprising shutting down a development instance.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which the invention pertains to modify the teaching of Gibiansky to stop model training based on a criterion as taught by Koch because the method of Gibiansky requires training of models and evaluation of the models, which means that training must be stopped at least temporarily. The specific training criteria taught by Koch would allow the user to have control over the computing resources used in the training process and terminating the instances after they finished performing the training task assigned to them would save compute resources.
Regarding claim 22, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the model task is at least one of a classification task, a prediction task, or a regression task. (Gibiansky, [0104]: classification and prediction. [0105]: regression)
Regarding claim 23, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the model storage comprises a cloud-based database. (Gibiansky, [0086] describes the memory occurring in the cloud (e.g. Amazon S3))
Regarding claim 24, the rejection of claim 21 is incorporated herein. Gibiansky does not appear to explicitly teach
wherein the index of stored models is configured to identify a suitable model based on the model task.
However, Chen—directed to analogous art--teaches
wherein the index of stored models is configured to identify a suitable model based on the model task. (Chen, Fig. 8 shows a method for accessing (i.e., retrieving) a machine learning model usage based at least in part on schema data. This is described at [0069-0074]. In particular, step 812 shows a step of accessing a machine learning model usage from element 316 (and more generally from the Machine learning server 320). [0042] indicates that the model usage includes the model and associated hyperparameters and task. The machine learning server is further described at [0039-0042]. ([0042] indicates that the model usage includes a description of task performed by the model, so this information is understood to be part of the index as described above.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have combined these references in this way for the same reasons given above with respect to claim 21.
Regarding claim 25, the rejection of claim 21 is incorporated herein. Gibiansky does not appear to explicitly teach
wherein retrieving the first model from the model storage is further based on a modeling characteristic, the modeling characteristic being at least one of a model type, a data schema, a training dataset type, or a training dataset identifier.
However, Chen—directed to analogous art--teaches
wherein retrieving the first model from the model storage is further based on a modeling characteristic, the modeling characteristic being at least one of a model type, a data schema, a training dataset type, or a training dataset identifier. (Chen, Fig. 8 shows a method for accessing (i.e., retrieving) a machine learning model usage based at least in part on schema data. This is described at [0069-0074]. In particular, step 812 shows a step of accessing a machine learning model usage from element 316. [0042] indicates that the model usage includes the model and associated hyperparameters and task. [0042] further indicates that the ML model may be selected based on a match to the historical interaction data.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have combined these references in this way for the same reasons given above with respect to claim 21.
Regarding claim 26, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the retrieved first hyperparameter is at least one of a training hyperparameter or an architectural hyperparameter. (Gibiansky, [0076] describes updating a partially or previously trained model. The “parameters” of Gibiansky are described at [0073-0078,0114] and include at least model type, “architectural hyperparameters” (e.g. [0075]: type of kernel and margin width; [0076]: number of trees), “training hyperparameters” (e.g., [0075]: whether or not to perform bagging), and values of model parameters (e.g. alpha coefficients in an SVM setting, see [0047]). As described at [0047], a model is identified with the parameter values which define it. [0078] describes retrieving this data from storage.)
Regarding claim 27, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the first development instance is further configured to retrieve the training data from a dataset generator or a training data database. ([0050] indicates that the optimization server 102 (i.e. development instance) may retrieve data from the data store 112. Since the data store 112 may include training data, it is understood to be a training data database.)
Regarding claim 28, the rejection of claim 21 is incorporated herein. Gibiansky does not appear to explicitly teach
wherein the training data comprises synthetic data generated by a generative adversarial network based on actual data.
However, Choi—directed to analogous art--teaches
wherein the training data comprises synthetic data generated by a generative adversarial network based on actual data. (Choi, Abstract describes using a Generative Adversarial Network (GAN) to generate synthetic data which may be used to perform predictive modeling tasks. Section 3 provides an overview of the method. Section 3.6., Presence disclosure, indicates that the synthetic data may be useful in preventing an attacker from determining that machine learning model was trained using a particular piece of data. Section 3.2. provides an overview of the GAN framework. In particular, the GAN is trained using actual data to generate synthetic data. That is, the synthetic data is generated based on the actual data. Appendix F discusses whether or not the data sets are dissimilar enough to fool an attacker. In particular, Figure 11 (and surrounding discussion) indicates that the attacker’s sensitivity is below 0.2 (with a Hamming distance of 0) and the precision is below 0.8 (with a Hamming distance of 0). Either of these indicates that a similarity between the synthetic data and actual data is less than a threshold (i.e., 0.2 or 0.8). Examiner notes that the claim requires that there be some similarity threshold for which the similarity between the synthetic data and actual data is less than that threshold, which is always true when the threshold is not predetermined.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have combined these references in this way for the same reasons given above with respect to claim 21.
Regarding claim 29, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the first performance metric depends on a similarity between data generated by the trained model and the training data. (Gibiansky, [0041] describes various fitness measures. In particular, error rate is the rate at which the training data is misclassified (i.e. the similarity between a prediction and the true classification/prediction/forecast).)
Regarding claim 30, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the termination condition is based on at least one of a threshold value of the first performance metric, an improvement rate of the first performance metric, a threshold value of the second performance metric, or an improvement rate of the second performance metric. (Gibiansky, [0086] indicates that the termination condition may be a number of iterations without the measure of fitness increasing (i.e., the improvement rate is too low). [0087] indicates that the accuracy may be compared to the accuracy of a competing model, so the threshold is the accuracy of the other model. This is performed for each of the models, so each of these would apply to either of the first or second performance metrics.)
Regarding claim 31, the rejection of claim 21 is incorporated herein. Furthermore Gibiansky teaches
storing the trained model in the model storage; and ([0050] indicates that the system includes a storage where trained parameters are stored. As described above, the parameters for a model specify the model. The storage is understood to be model storage by virtue of models being stored there.)
Gibiansky does not appear to explicitly teach
updating the index of stored models.
However, Chen—directed to analogous art--teaches
updating the index of stored models. ([0032] indicates that the model and feature extraction rule can be stored in the historical interaction dataset and used to respond to future requests for ML models.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which the invention pertains to modify the combination of Gibiansky to update the index of stored models as taught by Chen because this allows the model to be used to respond to future requests as described by Chen at [0032].
Regarding claim 32, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the operations further comprise, before determining whether the termination condition is satisfied: (Gibiansky, Figure 4, steps 306-310 form a loop in which parameters are selected, tested and refined. The loop occurs when the stop condition is not met.)
retrieving a second hyperparameter based on the first performance metric; (Gibiansky, Fig. 3, step 308 generates new parameter samples based on the new parameter distribution. See also [0078] for the generated samples being sent to the optimization unit. [0080-0082] indicates that a fitness score (i.e. performance metric) is used to generate the new parameter distribution. That means that the newly generated parameter is based on the fitness score.)
providing, to the first development instance, a command to generate a second trained model by training the first trained model based on the second hyperparameter; (Gibiansky, [0082] indicates that the new parameter is used to implement the model and that the first machine learning candidate may be sued to tune (i.e. train) the second candidate. Since the development instance performs this step and the development instance is computer-implemented, it is understood that a command for performing this step was provided to it.)
receiving, from the first development instance, the second trained model and a model output associated with the second trained model; and updating the first performance metric based on the model output. (Gibiansky, [0086] indicates that the process of generating models, evaluating them, updating the parameter distribution, and generating new parameter samples is iterated. [0086-0087] describe one of the stopping criteria being based on a measure of fitness (i.e. performance metric). [0091] describes the distributed scenario in which each optimization server manages data related to the model which it is optimizing. [0096] indicates that the coordination of the various optimization servers may be achieved by the data management unit 260. It is understood that for the data management unit 260 to manage this data, it must be received from the optimization servers. The use of the new performance metric is understood to be an update to the performance metric.)
Regarding claim 33, the rejection of claim 32 is incorporated herein. Furthermore, Gibiansky teaches
wherein the first hyperparameter is an architectural hyperparameter and the second hyperparameter is a training hyperparameter. (Gibiansky: The “parameters” of Gibiansky are described at [0073-0078,0114] and include at least model type, “architectural hyperparameters” (e.g. [0075]: type of kernel and margin width; [0076]: number of trees), “training hyperparameters” (e.g., [0075]: whether or not to perform bagging), and values of model parameters (e.g. alpha coefficients in an SVM setting, see [0047]). [0042] indicates that multiple parameters may be searched simultaneously. Since the models of Gibiansky include both architectural and training parameters, it is understood that both architectural and training parameters may be included at each iteration through the loop described above.)
Regarding claim 34, the rejection of claim 32 is incorporated herein. Furthermore, Gibiansky teaches
wherein retrieving the second hyperparameter is further based on a search strategy. (Gibiansky, [0082] indicates that new parameters are randomly generated and that the parameters determined to have the greatest potential to increase the measure of fitness are used. This is understood to constitute a search strategy.)
Regarding claim 35, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the operations further comprise performing, before determining whether the termination condition is satisfied: receiving a request to complete a new model task; (Gibiansky, Fig. 4, step 402 shows a step of receiving data. Fig. 5 provides an example of the GUI used by the users to input this data to the system. This example is described at [0104-0105]. In particular, the end of [0105] provides an example of a request input by a user. [0108] describes further details of commands which may be provided by experienced users. In particular, the user may identify details of the stop criterion including an iteration number. A specification of an iteration number greater than one is understood to be an indication that the model trained during a first pass should then be updated with new parameters. The training of a model in a second iteration is understood to be a new model task. This necessarily occurs before the termination condition is triggered.)
retrieving a second hyperparameter based on the first trained model and the new model task; (Gibiansky, [0076] describes updating a partially or previously trained model. The “parameters” of Gibiansky are described at [0073-0078,0114] and include at least model type, “architectural hyperparameters” (e.g. [0075]: type of kernel and margin width; [0076]: number of trees), “training hyperparameters” (e.g., [0075]: whether or not to perform bagging), and values of model parameters (e.g. alpha coefficients in an SVM setting, see [0047]). As described at [0047], a model is identified with the parameter values which define it. [0078] describes retrieving this data from storage. Because the abstract “model” is identified with the parameters which define it, the retrieval of these parameters is understood to be based on the model. It is understood that this step corresponds to a second (or subsequent) pass through the loop of Figure 3, steps 306-310.)
provisioning new computing resources to the first development instance, the first development instance being configured to train the first trained model based on the second hyperparameter and the new model task; (Gibiansky, Each of the optimization servers 102 is understood to be a development instance. [0093] indicates that this includes training the model based on retrieved data discussed earlier. See also Fig. 3, step 304 and [0098], where utilizing the parameters is understood to include training/updating the model. [0089] indicates that the resources devoted to a particular model instance may vary based on the likelihood that that model is optimal. That is, new computing resources may be provisioned to a development instance which is training a promising model.)
receiving, from the first development instance, a model output; and (Gibiansky, [0091] describes the distributed scenario in which each optimization server manages data related to the model which it is optimizing (i.e., a model output). [0096] indicates that the coordination of the various optimization servers may be achieved by the data management unit 260. It is understood that for the data management unit 260 to manage this data, it must be received from the optimization servers.)
updating the first performance metric based on the model output. (Gibiansky, [0086-0087] describe one of the stopping criteria being based on a measure of fitness. [0091] describes the distributed scenario in which each optimization server manages data related to the model which it is optimizing. [0096] indicates that the coordination of the various optimization servers may be achieved by the data management unit 260.)
Regarding claim 36, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
wherein the operations further comprise, before determining whether the termination condition is satisfied: receiving a request to complete a new model task; (Gibiansky, Fig. 4, step 402 shows a step of receiving data. Fig. 5 provides an example of the GUI used by the users to input this data to the system. This example is described at [0104-0105]. In particular, the end of [0105] provides an example of a request input by a user. [0108] describes further details of commands which may be provided by experienced users. In particular, the user may select a number and/or type of machine learning model. The selection of multiple models is understood to correspond to multiple model tasks (i.e. a model task and a new model task). This necessarily occurs before the termination condition is triggered)
retrieving a third model from the model storage ... (Gibiansky, [0076] describes updating a partially or previously trained model. The “parameters” of Gibiansky are described at [0073-0078,0114] and include at least model type, “architectural hyperparameters” (e.g. [0075]: type of kernel and margin width; [0076]: number of trees), “training hyperparameters” (e.g., [0075]: whether or not to perform bagging), and values of model parameters (e.g. alpha coefficients in an SVM setting, see [0047]). As described at [0047], a model is identified with the parameter values which define it. [0078] describes retrieving this data from storage. [0076] includes an example in which a model was trained on older data and is to be updated with newer data. This is understood to be an example in which the model is retrieved based on the task which the model performs (i.e. classifying emails).)
...retrieving a second hyperparameter based on the first trained model ... (Gibiansky, [0076] describes updating a partially or previously trained model. The “parameters” of Gibiansky are described at [0073-0078,0114] and include at least model type, “architectural hyperparameters” (e.g. [0075]: type of kernel and margin width; [0076]: number of trees), “training hyperparameters” (e.g., [0075]: whether or not to perform bagging), and values of model parameters (e.g. alpha coefficients in an SVM setting, see [0047]). As described at [0047], a model is identified with the parameter values which define it. [0078] describes retrieving this data from storage. Because the abstract “model” is identified with the parameters which define it, the retrieval of these parameters is understood to be based on the model. [0076] includes an example in which a model was trained on older data and is to be updated with newer data. This is understood to be an example in which the model is retrieved based on the task which the model performs (i.e. classifying emails). See also Fig. 3, step 302 and [0098].)
providing, to the first development instance, a command to train the third model based on the second hyperparameter; and (Gibiansky, [0093] indicates that the machine learning method is implemented and that a model is trained. [0092] describes generating new parameter configurations. Since a model is specified by its parameters, it is understood that implanting a machine learning method according to a set of parameters is creating a model instance. As described at [0092], this is performed by the optimization server 102. See also Fig. 3, step 304 and [0098], where utilizing the parameters is understood to include training/updating the model. Since, in the embodiment considered in this rejection, this occurs on the cloud, it is understood that a command to perform this must have been transmitted.)
receiving a third performance metric from the first development instance, wherein determining whether the termination condition is satisfied is further based on the third performance metric. (Gibiansky, [0086-0087] describe one of the stopping criteria being based on a measure of fitness. [0091] describes the distributed scenario in which each optimization server manages data related to the model which it is optimizing. [0096] indicates that the coordination of the various optimization servers may be achieved by the data management unit 260. It is understood that for the data management unit 260 to manage this data, it must be received from the optimization servers.)
Gibiansky does not appear to explicitly teach
retrieving a third model from the model storage based on the new model task;
retrieving a second hyperparameter based on the first trained model and the new model task;
However, Chen—directed to analogous art--teaches
retrieving a third model from the model storage based on the new model task; retrieving a second hyperparameter based on the first trained model and the new model task; (Chen, Fig. 8 shows a method for accessing (i.e., retrieving) a machine learning model based at least in part on schema data. This is described at [0069-0074]. In particular, step 812 shows a step of accessing a machine learning model usage from element 316 (and more generally from the Machine learning server 320). [0042] indicates that the model usage includes the model and associated hyperparameters and task. The machine learning server is further described at [0039-0042]. [0042] indicates that the model usage includes a description of task performed by the model, so this information is understood to be part of the index as described above. [0045] and Figure 3 and indicates that the models may be updated/retrained, which is based on the selecting described at [0042] which is based at least in part on the task.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have combined these references in this way for the same reasons given above with respect to claim 21.
Claim 37 recites a method which is substantially similar to the steps performed by the system of claim 21. Claim 37 is rejected with same rationale.
Regarding claim 38, the rejection of claim 21 is incorporated herein. Furthermore, Gibiansky teaches
memory for storing instructions; and one or more processors configured to execute the stored instructions to perform first operations comprising (Gibiansky, [0010] describes an implementation as a system (understood to be a model optimizer) comprising memory storing instructions and processors.)
Gibiansky does not appear to explicitly teach
a dataset generator ... generating training data based on actual data; and
However, Choi—directed to analogous art--teaches
a dataset generator comprising: ... generating training data based on actual data; and (Choi, Abstract describes using a Generative Adversarial Network (GAN) to generate synthetic data which may be used to perform predictive modeling tasks. Section 3 provides an overview of the method. Section 3.6., Presence disclosure, indicates that the synthetic data may be useful in preventing an attacker from determining that machine learning model was trained using a particular piece of data. Section 3.2. provides an overview of the GAN framework. In particular, the GAN is trained using actual data to generate synthetic data. That is, the synthetic data is generated based on the actual data. Appendix F discusses whether or not the data sets are dissimilar enough to fool an attacker. In particular, Figure 11 (and surrounding discussion) indicates that the attacker’s sensitivity is below 0.2 (with a Hamming distance of 0) and the precision is below 0.8 (with a Hamming distance of 0). Either of these indicates that a similarity between the synthetic data and actual data is less than a threshold (i.e., 0.2 or 0.8). Examiner notes that the claim requires that there be some similarity threshold for which the similarity between the synthetic data and actual data is less than that threshold, which is always true when the threshold is not predetermined.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have combined these references in this way for the same reasons given above with respect to claim 21.
Regarding claim 39, the rejection of claim 38 is incorporated herein. Gibiansky does not appear to explicitly teach
wherein the training data comprises synthetic data generated by a generative adversarial network based on the actual data.
However, Choi—directed to analogous art--teaches
wherein the training data comprises synthetic data generated by a generative adversarial network based on the actual data. (Choi, Abstract describes using a Generative Adversarial Network (GAN) to generate synthetic data which may be used to perform predictive modeling tasks. Section 3 provides an overview of the method. Section 3.6., Presence disclosure, indicates that the synthetic data may be useful in preventing an attacker from determining that machine learning model was trained using a particular piece of data. Section 3.2. provides an overview of the GAN framework. In particular, the GAN is trained using actual data to generate synthetic data. That is, the synthetic data is generated based on the actual data. Appendix F discusses whether or not the data sets are dissimilar enough to fool an attacker. In particular, Figure 11 (and surrounding discussion) indicates that the attacker’s sensitivity is below 0.2 (with a Hamming distance of 0) and the precision is below 0.8 (with a Hamming distance of 0). Either of these indicates that a similarity between the synthetic data and actual data is less than a threshold (i.e., 0.2 or 0.8). Examiner notes that the claim requires that there be some similarity threshold for which the similarity between the synthetic data and actual data is less than that threshold, which is always true when the threshold is not predetermined.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have combined these references in this way for the same reasons given above with respect to claim 21.
Claim 40 is rejected under 35 U.S.C. 103 as being unpatentable over “Gibiansky” (US 2016/0110657 A1) in view of “Chen” (US 2019/0362222 A1), further in view of “Choi” (Generating Multi-label Discrete Patient Records using Generative Adversarial Networks), further in view of “Koch” (US 2018/0240041 A1), and further in view of “Spertus” (US 2017/0149793 A1).
Regarding claim 40, the rejection of claim 38 is incorporated herein. Gibiansky does not appear to explicitly teach
identifying at least one sensitive data item in the actual data; and generating at least one synthetic data item to replace the at least one sensitive data item in the actual data.
However, Choi—directed to analogous art--teaches
generating at least one synthetic data item to replace the at least one sensitive data item in the actual data. (Choi, Abstract describes using synthetic EHR data in place of EHR data (i.e., all of the data records in the EHR data is replaced by synthetic data). Section 3 provides implementation details.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have combined these references in this way for the same reasons given above with respect to claim 21.
The combination of Gibiansky, Chen, Choi, Koch and does not appear to explicitly teach
identifying at least one sensitive data item in the actual data; and
However, Spertus—directed to analogous art--teaches
identifying at least one sensitive data item in the actual data; and (Spertus, [0006] describes determining/identifying a data field that contains sensitive data and anonymizing that data. In the combination with Choi, the anonymization technique taught by Choi would be used in place of the proposed anonymization techniques in Spertus.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Gibiansky, Chen, Choi and Koch to identify sensitive data as taught by Spertus because the techniques of Spertus address a need for additional and improved systems and methods for anonymizing log entries as described by Spertus at [0002].
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21-39 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 8, 9, 10, 1, 4, , 14, 1, 13, 15, 2, 3, 4, 5, 6, 7, 17, 18, and 18, respectively (see also table below), of U.S. Patent No. US 11,256,555 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because they are broader than the patented claims as shown in the table below. In the table below, substantially similar subject matter is shown in bold.
Instant Application
US 11,256,555 B2 Patent
21. (New) A scalable system for completing a model task using a serverless architecture, the system comprising:
a model optimizer comprising:
memory for storing instructions; and
one or more processors configured to execute the stored instructions to perform operations comprising:
receiving a request to complete a model task;
retrieving a first model from a model storage by selecting, based on the model task, the first model in an index of stored models;
retrieving a first hyperparameter based on the first model and the model task;
provisioning computing resources to a first development instance configured to train the first model based on the first hyperparameter and the model task;
creating an instance of the first model from the first development instance;
retrieving, by the first development instance, training data comprising synthetic data based on actual data;
training the instance of the first model by the first development instance using the training data to obtain a first trained model;
terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a dataset generated by the trained model and a reference dataset, a data similarity score between the dataset generated by the trained model and the reference dataset, or a data quality score for the dataset generated by the trained model;
generating, by the first development instance, a first performance metric of the trained model;
receiving, from the first development instance, the trained model and the first performance metric;
receiving, from a second development instance, a second performance metric associated with a second model;
determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and
terminating the first development instance based on the determination that the termination condition is satisfied.
1. A scalable system for completing a model task using a serverless architecture, the system comprising:
a model optimizer, wherein the model optimizer comprises:
one or more memory units for storing instructions; and
one or more processors configured to execute the stored instructions to perform operations comprising:
receiving a request to complete a model task, the request comprising a received data schema of a dataset, the received data schema comprising data structure descriptors;
retrieving a first model from a model storage by selecting the first model in an index of stored models, the first model being configured to generate synthetic data satisfying a stored data schema comprising data structure descriptors, the retrieving being based on the model task and a similarity metric indicating a number of mismatches between the data structure descriptors of the received data schema and the data structure descriptors of the stored data schema;
retrieving a hyperparameter based on the first model and the model task;
spinning up a first development instance of the serverless architecture, the first development instance being configured to train the first model based on the retrieved hyperparameter and the model task;
creating, by the first development instance, a model instance of the first model;
training the model instance by the development instance using training data to obtain a trained model, the training comprising a hyperparameter search based on the retrieved hyperparameter, wherein the training data comprises synthetic data generated by a generative adversarial network using actual data, wherein the generative adversarial network is distinct from the first model;
terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a synthetic dataset generated by the trained model and a reference dataset, a data similarity score between the synthetic dataset and the reference dataset, or a data quality score based on the synthetic dataset and the reference dataset;
generating, by the first development instance, a first performance metric of the trained model;
receiving, from the first development instance, the trained model and the first performance metric;
receiving, from a second development instance, a second performance metric associated with a second model;
determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and
terminating the first development instance based on the determination that the termination condition is satisfied.
22. (New) The scalable system of claim 21, wherein the model task is at least one of a classification task, a prediction task, or a regression task.
8. The scalable system of claim 1, where in the model task is at least one of a classification task, a prediction task, or a regression task.
23. (New) The scalable system of claim 21, wherein the model storage comprises a cloud-based database.
9. The scalable system of claim 1, wherein the model storage comprises a cloud-based database.
24. (New) The scalable system of claim 21, wherein the index of stored models is configured to identify a suitable model based on the model task.
10. The scalable system of claim 1, wherein the index of stored models is configured to identify a suitable model based on the model task.
25. (New) The scalable system of claim 21, wherein retrieving the first model from the model storage is further based on a modeling characteristic, the modeling characteristic being at least one of a model type, a data schema, a training dataset type, or a training dataset identifier.
1. ... retrieving a first model from a model storage by selecting the first model in an index of stored models, the first model being configured to generate synthetic data satisfying a stored data schema comprising data structure descriptors, the retrieving being based on the model task and a similarity metric indicating a number of mismatches between the data structure descriptors of the received data schema and the data structure descriptors of the stored data schema;
26. (New) The scalable system of claim 21, wherein the retrieved first hyperparameter is at least one of a training hyperparameter or an architectural hyperparameter.
4. The scalable system of claim 3, wherein the first hyperparameter is an architectural hyperparameter and the second hyperparameter is a training hyperparameter.
27. (New) The scalable system of claim 21, wherein the first development instance is further configured to retrieve the training data from a dataset generator or a training data database.
14. The scalable system of claim 1, wherein the development instance is further configured to retrieve the training data from a dataset generator or a training data database.
28. (New) The scalable system of claim 21, wherein the training data comprises synthetic data generated by a generative adversarial network based on actual data.
1. ...wherein the training data comprises synthetic data generated by a generative adversarial network using actual data,
29. (New) The scalable system of claim 21, wherein the first performance metric depends on a similarity between data generated by the trained model and the training data.
13. The scalable system of claim 1, wherein the performance metric depends on a similarity between data generated by the trained model and training data.
30. (New) The scalable system of claim 21, wherein the termination condition is based on at least one of a threshold value of the first performance metric, an improvement rate of the first performance metric, a threshold value of the second performance metric, or an improvement rate of the second performance metric.
15. The scalable system of claim 1, wherein the termination condition is based on at least one of a threshold value of the performance metric or an improvement rate of the performance metric.
31. (New) The scalable system of claim 21, the operations further comprising:
storing the trained model in the model storage; and
updating the index of stored models.
2. The scalable system of claim 1, the operations further comprising:
storing the trained model in the model storage; and
updating the index of stored models.
32. (New) The scalable system of claim 21, wherein the operations further comprise, before determining whether the termination condition is satisfied:
retrieving a second hyperparameter based on the first performance metric;
providing, to the first development instance, a command to generate a second trained model by training the first trained model based on the second hyperparameter;
receiving, from the first development instance, the second trained model and a model output associated with the second trained model; and
updating the first performance metric based on the model output.
3. The scalable system of claim 1, wherein:
the retrieved hyperparameter is a first hyperparameter,
the trained model is a first trained model, and
before determining whether the termination condition is satisfied, the operations further comprise:
retrieving a second hyperparameter based on the first performance metric;
providing, to the first development instance, a command to generate a second trained model by training the first trained model based on the second hyperparameter;
receiving, from the first development instance, the second trained model and a model output associated with the second trained model; and
updating the first performance metric based on the model output.
33. (New) The scalable system of claim 32, wherein the first hyperparameter is an architectural hyperparameter and the second hyperparameter is a training hyperparameter.
4. The scalable system of claim 3, wherein the first hyperparameter is an architectural hyperparameter and the second hyperparameter is a training hyperparameter.
34. (New) The scalable system of claim 32, wherein retrieving the second hyperparameter is further based on a search strategy.
5. The scalable system of claim 3, wherein retrieving the second hyperparameter is further based on a search strategy.
35. (New) The scalable system of claim 21, wherein the operations further comprise performing, before determining whether the termination condition is satisfied:
receiving a request to complete a new model task;
retrieving a second hyperparameter based on the first trained model and the new model task;
provisioning new computing resources to the first development instance, the first development instance being configured to train the first trained model based on the second hyperparameter and the new model task;
receiving, from the first development instance, a model output; and
updating the first performance metric based on the model output.
6. The scalable system of claim 1, wherein, before determining whether the termination condition is satisfied, the operations further comprise:
receiving a request to complete a new model task;
retrieving a new hyperparameter based on the trained model and the new model task;
provisioning new computing resources to the development instance, the development instance configured to train the trained model based on the new hyperparameter and the new model task; and
receiving, from the first development instance, a model output; and
updating the first performance metric based on the model output.
36. (New) The scalable system of claim 21, wherein the operations further comprise, before determining whether the termination condition is satisfied:
receiving a request to complete a new model task;
retrieving a third model from the model storage based on the new model task;
retrieving a second hyperparameter based on the first trained model and the new model task;
providing, to the first development instance, a command to train the third model based on the second hyperparameter; and
receiving a third performance metric from the first development instance, wherein determining whether the termination condition is satisfied is further based on the third performance metric.
7. The scalable system of claim 1, wherein, before determining whether the termination condition is satisfied, the operations further comprise:
receiving a request to complete a new model task;
retrieving a new model from the model storage based on the model task and the index of stored models;
retrieving a new hyperparameter based on the trained model and the new model task;
providing, to the first development instance, a command to train the new model based on the new hyperparameter; and
receiving a third performance metric from the first development instance,
wherein determining whether the termination condition is satisfied is further based on the third performance metric.
37. (New) A method for completing a model task using a serverless architecture comprising:
receiving a request to complete a model task;
retrieving a first model from a model storage by selecting the first model in an index of stored models based on the model task;
retrieving a hyperparameter based on the first model and the model task;
provisioning computing resources to a first development instance configured to train the first model based on the retrieved hyperparameter and the model task;
creating an instance of the first model from the first development instance;
retrieving, by the first development instance, training data comprising synthetic data based on actual data;
training the instance of the first model by the first development instance using the training data to obtain a trained model;
terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a dataset generated by the trained model and the reference dataset, a data similarity score between the dataset generated by the trained model and the reference dataset, or a data quality score for the dataset generated by the trained model;
generating, by the first development instance, a first performance metric of the trained model;
receiving, from the first development instance, the trained model and the first performance metric;
receiving, from a second development instance, a second performance metric associated with a second model;
determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and
terminating the first development instance based on the determination that the termination condition is satisfied.
17. A method for completing a model task using a serverless architecture comprising:
receiving a request to complete a model task, the request comprising a received data schema of a dataset, the received data schema comprising data structure descriptors;
retrieving a first model from a model storage by selecting the first model in an index of stored models, the index comprising a stored data schema, the stored data schema comprising data structure descriptors, wherein the first model is configured to generate data satisfying the stored data schema, the retrieving being based on the model task and a similarity metric indicating a number of mismatches between the data structure descriptors of the received data schema and the data structure descriptors of the stored data schema;
retrieving a hyperparameter based on the first model and the model task;
spinning up a first development instance of the serverless architecture, the development instance being configured to train the first model based on the retrieved hyperparameter and the model task;
creating, by the first development instance, a model instance of the retrieved model;
training the model instance by the development instance using training data to obtain a trained model, the training comprising a hyperparameter search based on the retrieved hyperparameter, wherein the training data comprises synthetic data generated by a generative adversarial network using actual data, wherein the generative adversarial network is distinct from the first model;
terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a synthetic dataset generated by the trained model and a reference dataset, a data similarity score between the synthetic dataset and the reference dataset, or a data quality score based on the synthetic dataset and the reference dataset;
generating, by the first development instance, a first performance metric of the trained model;
receiving, from the development instance, the trained model and the first performance metric of the trained model;
receiving, from a second development instance, a second performance metric associated with a second model;
determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and
terminating the first development instance based on the determination that the termination condition is satisfied.
38. (New) A scalable system for completing a model task using a serverless architecture, the system comprising:
a dataset generator comprising:
memory for storing instructions; and
one or more processors configured to execute the stored instructions to perform first operations comprising generating training data based on actual data; and
a model optimizer comprising:
memory for storing instructions; and
one or more processors configured to execute the stored instructions to perform second operations comprising:
receiving a request to complete a model task;
retrieving a first model from a model storage by selecting the first model in an index of stored models based on the model task;
retrieving a hyperparameter based on the first model and the model task;
provisioning computing resources to a first development instance configured to train the first model based on the retrieved hyperparameter and the model task;
creating an instance of the first model from a first development instance;
training the instance of the first model by the first development instance using the training data to obtain a trained model;
terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a dataset generated by the trained model and the reference dataset, a data similarity score between the dataset generated by the trained model and the reference dataset, or a data quality score for the dataset generated by the trained model;
generating, by the first development instance, a first performance metric of the trained model;
receiving, from the first development instance, the trained model and the first performance metric;
receiving, from a second development instance, a second performance metric associated with a second model;
determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and
terminating the first development instance based on the determination that the termination condition is satisfied.
18. A scalable system for completing a model task using a serverless architecture, the system comprising:
a model optimizer, wherein the model optimizer comprises:
one or more memory units for storing instructions; and
one or more processors configured to execute the stored instructions to perform operations comprising:
receiving a request to complete a model task, the request comprising a received data schema of a dataset, the data schema including data structure descriptors of the received data schema;
retrieving a first model from a model storage by selecting the retrieved model in an index of stored models, the index comprising a stored data schema including data structure descriptors of a stored dataset, and the first model being configured to generate synthetic data satisfying the stored data schema, the retrieving being based on the model task and a similarity metric indicating a number of mismatches between the data-structure descriptors of the received data schema and the data-structure descriptors of the stored data schema;
retrieving a hyperparameter based on the first model and the model task;
spinning up a first development instance of the serverless architecture, the first development instance being configured to train the first model based on the retrieved hyperparameter and the model task;
creating, by the first development instance, a model instance of the first model;
training the model instance by the first development instance using training data, the training comprising a hyperparameter search based on the retrieved hyperparameter, wherein the training data comprises synthetic data generated by a generative adversarial network using actual data, wherein the generative adversarial network is distinct from the first model;
terminating, by the first development instance, model training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a synthetic dataset generated by the trained model and a reference dataset, a data similarity score between the synthetic dataset and the reference dataset, or a data quality score based on the synthetic dataset and the reference dataset;
generating, by the first development instance, a first performance metric of the trained model;
receiving, from the first development instance, the trained model and a first performance metric associated with the trained model;
receiving, from a second development instance, a second performance metric associated with a different model;
determining that a termination condition is satisfied based on at least one of the second performance metric or the second performance metric; and
terminating the development instance based on the determination that the termination condition is satisfied.
39. (New) The system of claim 38, wherein the training data comprises synthetic data generated by a generative adversarial network based on the actual data.
18... wherein the training data comprises synthetic data generated by a generative adversarial network using actual data
Claim 40 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 18 of U.S. Patent No. US 11,256,555 B2 in view of “Spertus” (US 2017/0149793 A1).
Regarding claim 40, the rejection of claim 38 is incorporated herein. Patented claim 18 does not appear to explicitly teach
identifying at least one sensitive data item in the actual data; and generating at least one synthetic data item to replace the at least one sensitive data item in the actual data.
However, Spertus—directed to analogous art--teaches
identifying at least one sensitive data item in the actual data; and generating at least one synthetic data item to replace the at least one sensitive data item in the actual data. (Spertus, [0006] describes determining/identifying a data field that contains sensitive data and anonymizing that data. In the combination with Choi, the anonymization technique taught by Choi would be used in place of the proposed anonymization techniques in Spertus.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified patented claim 18 to identify sensitive data as taught by Spertus because the techniques of Spertus address a need for additional and improved systems and methods for anonymizing log entries as described by Spertus at [0002].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Park (Data Synthesis based on Generative Adversarial Networks) – Abstract describes using a GAN to replace to sensitive data by synthesizing fake tables.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Markus A Vasquez whose telephone number is (303)297-4432. The examiner can normally be reached Monday to Friday 9AM to 4PM PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached on (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARKUS A. VASQUEZ/ Primary Examiner, Art Unit 2121
Prosecution Insights