DETAILED ACTION
Acknowledgements
This Office Action is in response to Applicant’s response/application filed on 04/19/2024.
The Examiner notes that citations to United States Patent Application Publication paragraphs are formatted as [####], #### representing the paragraph number.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-11 are currently pending and have been examined.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 10 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 10 recites the limitations: “the pre-ML score”. There is insufficient antecedent basis for this limitation in the claim. For examination purposes examiner has interpreted “the pre-ML score” to be “a pre-ML score”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
As per claims 1-20, the claimed invention is directed to an abstract idea without significantly more because:
• Claim 1 recites:
obtaining an empirically collected dataset;
generating a synthetic dataset based on the empirically collected dataset;
performing an automated machine learning (Auto-ML) analysis of the synthetic dataset; and
generating an Auto-ML score of the synthetic dataset based on a result of the Auto-ML analysis.
• Under Step 1 of the Section 101 analysis, the claim(s) is/are directed to a method, a system, and a manufacture, which are statutory categories of invention.
• Under Step 2A Prong One of the 2019 Revised Patent Subject Matter Eligiblity Guidance, the claimed invention as drafted includes language (see underlined language above) that recites an abstract idea of generating synthetic dataset and scoring the synthetic dataset based on an analysis (a certain method of organizing human activity) but for the recitation of additional claim elements. That is, other than reciting “automated machine learning (Auto-ML)”, nothing in the claim precludes the language from being considered as performed by a person.
• Under Step 2A Prong Two of the 2019 Revised Patent Subject Matter Eligiblity Guidance, the additional claim element(s), considered individually, do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception and in a manner that integrates the exception into a practical application of the exception. The additional claim elements(s) such as “automated machine learning (Auto-ML)”, generally link the use of the judicial exception to a particular technological environment or field of use of artificial intelligence.
• Under Step 2A Prong Two, the additional claim element(s), considered in combination, do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception and in a manner that integrates the exception into a practical application of the exception. The combination of elements is no more than the sum of their parts. Unlike the eligible claims in Diehr and Bascom, in which the elements limiting the exception taken together improve a technical field, the instant claim lacks an improvement to the functioning of a computer or to any other technology or technical field.
• Under Step 2B, the additional claim element(s), considered individually and in combination, do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself for similar reasons outlined under Step 2A Prong Two.
A similar analysis can be applied to dependent claims 3, 5, 6, 9, 10, 11 which further recite the abstract idea of identifying an optimal dataset, validating the datasets, generating a score based on statistical comparison, splitting collected datasets into a plurality of classification groups, and applying a synthetic data generate method to the classification groups (a certain method of organizing human activity). That is, other than reciting the additional elements, nothing in the claim precludes the language from being considered as performed by a person.
A similar analysis can be applied to dependent claims 2, 4, 7, 8 which include additional claim elements that merely add the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. For example, “Auto-ML”, “training…” generally link the use of the judicial exception to a particular technological environment or field of use of artificial intelligence.
Therefore, claims 1-11 are rejected under 35 U.S.C. §101.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hameed (“Improving mortality prediction in Acute Pancreatitis by machine learning and data augmentation”, M. Asad Bin Hameed, Computers in Biology and Medicine 150 (2022) 106077, Sep 11, 2022).
Regarding claim(s) 1, Hameed discloses:
obtaining an empirically collected dataset (By disclosing, “The data is extracted from MIMIC III and MIMIC IV datasets using the big data framework, the Big Query. The data extraction pipelineis the same for both the MIMIC data sets. We extracted the lab-eventstable and joined it to the lab-items table through BigQuery.” (Section 3.2));
generating a synthetic dataset based on the empirically collected dataset (By disclosing, synthetic data generation via SMOTE, GAN, and VAE (Section 3.4)) ;
performing an automated machine learning (Auto-ML) analysis of the synthetic dataset (By disclosing, “The sklearn library was used to implement RF, DT, and LR classifiers, while Keras was used to implement the ANN and DNN. The XGBoost classifier was run using the XGboost library. Each experiment was executed 10 times using stratified sampling, and the metrics were reported as an average of these runs.” (Section 4.1)); and
generating an Auto-ML score of the synthetic dataset based on a result of the Auto-ML analysis. (By disclosing, “𝐹𝛽 is a variation of the F1 score that allows prioritizing one metric over the other.” (Section 4.2); and “The best results were achieved by the Random Forest on the synthetic data generated by CTGAN with a 50 to 50 class split. It managed to get the highest 𝐹𝛽 score of 0.702. The recall and AUC values were significantly high, too, with 0.833 and 0.930, respectively. RF also performed quite well on 70 to 30 upsampling by CTGAN, with 0.675 𝐹𝛽 and 0.943 AUC. The performance of RF on the upsampled data using a variational autoencoder, TVAE, was reasonably close to CTGAN. It achieved a 0.698 𝐹𝛽 score and 0.948 AUC on a 50 to 50 class split. Other classifiers like XG-boost, MLP, and DNN also perform well on TVAE upsampled data. However, this was not true in the case of CTGAN. The performance of classifiers on upsampled datasets by CTAB, Copulgan, and TGAN was reasonable but less than that of CTGAN. Random Forest classifier was a clear winner as it performed well on almost all upsampled datasets.” (Section 5, Table 6)).
Regarding claim(s) 2, Hameed discloses:
generating a plurality of synthetic datasets according to a plurality of different synthetic data generation methods, wherein the Auto-ML analysis is performed, and the Auto-ML score is generated, for each of the plurality of synthetic datasets. (By disclosing, “We propose the notion of using advanced tabular generative adversarial networks (GAN) and variational autoencoders (VAE) for synthesizing AP data to eliminate class imbalance and data scarcity issues. The five state-of-the-art generative models: CTGAN, TGAN, CTAB,
and CopulaGAN, and TVAE are employed to generate multiple synthetic datasets with two majority to minority class splits: a 50 to 50 split and a 70 to 30 class split. In addition to GANS and TVAE, SMOTE technique is also used to generate upsampled datasets with the two class splits.” (Section 1, Section 3.4, Table 6)).
Regarding claim(s) 3, Hameed discloses:
wherein the plurality of different synthetic data generation methods comprise single function and multi-function models. (By disclosing, “GANs, generative adversarial networks, are emerging ML models that can generate synthetic data with the same statistics as the given training data. A GAN consists of a generator network that produces synthetic (fake) data records and a discriminator network that uses the real data along with generated one to determine which data records are fake or synthesized. These two are coupled by loss function to play a min–max game that is adversarial in nature” (Section 3.4); and “Different functions: tanh, relu, softmax, and sigmoid are tested on both these models. The best results are reported in the experiment sections.” (Section 3.5.1)).
Regarding claim(s) 5, Hameed discloses:
identifying an optimal one of the plurality of synthetic datasets based on the Auto-ML scores. (By disclosing, “The best results were achieved by the Random Forest on the synthetic data generated by CTGAN with a 50 to 50 class split. It managed to get the highest 𝐹𝛽 score of 0.702. The recall and AUC values were significantly high, too, with 0.833 and 0.930, respectively. RF also performed quite well on 70 to 30 upsampling by CTGAN, with 0.675 𝐹𝛽 and 0.943 AUC. The performance of RF on the upsampled data using a variational autoencoder, TVAE, was reasonably close to CTGAN. It achieved a 0.698 𝐹𝛽 score and 0.948 AUC on a 50 to 50 class split. Other classifiers like XG-boost, MLP, and DNN also perform well on TVAE upsampled data. However, this was not true in the case of CTGAN. The performance of classifiers on upsampled datasets by CTAB, Copulgan, and TGAN was reasonable but less than that of CTGAN. Random Forest classifier was a clear winner as it performed well on almost all upsampled datasets.” (Section 5, Table 6)).
Regarding claim(s) 6, Hameed discloses:
validating the plurality of synthetic datasets based on the Auto-ML scores. (By disclosing, “We propose the notion of using advanced tabular generative adversarial networks (GAN) and variational autoencoders (VAE) for synthesizing AP data to eliminate class imbalance and data scarcity issues. The five state-of-the-art generative models: CTGAN, TGAN, CTAB, and CopulaGAN, and TVAE are employed to generate multiple synthetic datasets with two majority to minority class splits: a 50 to 50 split and a 70 to 30 class split. In addition to GANS and TVAE, SMOTE technique is also used to generate upsampled datasets with the two class splits. This research ensures that the data used is not only of reasonable size but includes diverse medical cases to train classifiers properly and overcome the issues of overfitting and underfitting in ML models. Hence, the reported accuracy and analysis are correct, not just false infatuation and results.” (Section 1)).
Regarding claim(s) 11, Hameed discloses:
wherein generating the synthetic dataset comprises: splitting the empirically collected dataset into a plurality of classification groups; and applying a synthetic data generation method to each of the plurality of classification groups (Section 4.5.3, Table 12).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hameed (“Improving mortality prediction in Acute Pancreatitis by machine learning and data augmentation”, M. Asad Bin Hameed, Computers in Biology and Medicine 150 (2022) 106077, Sep 11, 2022), in view of Narayanan (US 20240249158).
Regarding claim(s) 4, Hameed discloses:
wherein the plurality of synthetic data generation methods comprise single function and multi-function versions of each of copula-GAN, CT-GAN, and TVAE models. (By disclosing, “We propose the notion of using advanced tabular generative adversarial networks (GAN) and variational autoencoders (VAE) for synthesizing AP data to eliminate class imbalance and data scarcity issues. The five state-of-the-art generative models: CTGAN, TGAN, CTAB, and CopulaGAN, and TVAE are employed to generate multiple synthetic datasets with two majority to minority class splits: a 50 to 50 split and a 70 to 30 class split. In addition to GANS and TVAE, SMOTE technique is also used to generate upsampled datasets with the two class splits. This research ensures that the data used is not only of reasonable size but includes diverse medical cases to train classifiers properly and overcome the issues of overfitting and underfitting in ML models. Hence, the reported accuracy and analysis are correct, not just false infatuation and results.” (Section 1)).
Hameed does not disclose, but Narayanan teaches:
Gaussian copula (By disclosing, “Other tabular synthetic data generation techniques may include GaussianCopula, CopulaGAN, triplet-based variational autoencoder (TVAE) and/or the like.” ([0099] of Narayanan)).
Therefore, it would have been obvious to one of ordinary skill in the art at the effective filing date of the present application to modify the invention of Hameed in view of Narayanan to include Gaussian copula as a synthetic data generation method. Doing so would result in an improved invention because this would leverages the advantages of using Gaussian copula (e.g. Flexibility in Modeling Dependencies, Separation of Marginals and Dependent Structure, Mathematical Tractability, etc.).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hameed (“Improving mortality prediction in Acute Pancreatitis by machine learning and data augmentation”, M. Asad Bin Hameed, Computers in Biology and Medicine 150 (2022) 106077, Sep 11, 2022), in view of Soni (US 20200134446).
Regarding claim(s) 7, Hameed does not disclose, but Soni teaches:
training a synthetic data machine learning system with a training portion of the synthetic dataset (By disclosing, “The example system includes a synthetic data generator to generate first synthetic data from the first real data, the first synthetic data to be used by the DNN generator to generate a second DNN model.” ([0004] of Soni));
training a real data machine learning system with a training portion of the empirically collected dataset (By disclosing, “The example system includes a deep neural network (DNN) generator to generate a first DNN model using first real data.” ([0004] of Soni));
inputting a testing portion of the synthetic dataset to the trained synthetic data machine learning system (By disclosing, “The example system includes an artificial intelligence model deployment processor to deploy an artificial
intelligence model trained and tested using the synthetic data set.” ([0004], [0104] of Soni));
inputting a testing portion of the empirically collected dataset to the trained real data machine learning system (By disclosing, “At block 1408, the first
trained network and second trained network are compared to evaluate their performance. For example, the evaluator 1020, 1060, 1325 compares the first
and second trained DNNs 1010, 1018, 1050, 1058 to determine whether the
first network trained on real data is indistinguishable and/or otherwise comparable to the second network trained on synthetic data (e.g., accuracy, precision, consistency, etc.).” ([0104] of Soni));
inputting the testing portion of the empirically collected dataset to the trained synthetic data machine learning system (By disclosing, “One approach to a scalable solution is to train/retrain one A.I. model for each medical condition at every site in a sequence as shown in FIG. 9. For example, as shown in FIG. 9, a pretrained DNN 902 is provided to a first site 910, which uses its data 920 to retrain 930 the DNN to produce a first DNN model 904. The retrained DNN model 904 is provided to a second site 912, which uses its data 922 to retrain 932 the DNN 904 to produce a second DNN model 906. The retrained DNN model 906 is provided to a third site 914, which uses its data 924 to retrain 934 the DNN 906 to produce a third DNN model 908. This process of site-based retraining can be repeated for one or more sites that have data available for retraining the network and/or until a certain reliability/accuracy threshold is reached, for example. Thus, the model trained at a first site using that site's region-specific patient data can be retrained at a second site.” ([0077]-[0078] of Soni)),
wherein the Auto-ML score is based on a comparison of the outputs of the trained real data and synthetic data machine learning systems (By disclosing, “At block 1408, the first trained network and second trained network are compared to evaluate their performance. For example, the evaluator 1020, 1060, 1325
compares the first and second trained DNNs 1010, 1018, 1050, 1058 to determine whether the first network trained on real data is indistinguishable and/or otherwise comparable to the second network trained on synthetic data (e.g., accuracy, precision, consistency, etc.).” ([0104] of Soni); and “In certain examples, the evaluator 1325 can compare two or more DNN models to determine whether the output of each DNN model matches, is within a margin of error/standard deviation/tolerance, etc.” ([0095] of Soni)).
. Therefore, it would have been obvious to one of ordinary skill in the art at the effective filing date of the present application to modify the invention of Hameed in view of Soni to include techniques of training a synthetic data machine learning system with a training portion of the synthetic dataset; training a real data machine learning system with a training portion of the empirically collected dataset; inputting a testing portion of the synthetic dataset to the trained synthetic data machine learning system; inputting a testing portion of the empirically collected dataset to the trained real data machine learning system; and inputting the testing portion of the empirically collected dataset to the trained synthetic data machine learning system, wherein the Auto-ML score is based on a comparison of the outputs of the trained real data and synthetic data machine learning systems. Doing so would result in an improved invention because this would allow the system to check whether the generated synthetic machine learning model and the synthetic data preserves useful behavior.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hameed (“Improving mortality prediction in Acute Pancreatitis by machine learning and data augmentation”, M. Asad Bin Hameed, Computers in Biology and Medicine 150 (2022) 106077, Sep 11, 2022), in view of Beigi (EP 4137980 A1).
Regarding claim(s) 9, Hameed does not disclose, but Beigi teaches:
generating a pre-ML score based on a statistical comparison of the empirically collected dataset and the synthetic dataset (By disclosing, “To evaluate the fidelity of the synthesized dataset, cross-validation tests were performed on both the synthesized and source dataset to examine how well the source's underlying properties are preserved. The Fisher Exact and the Kolmogorov-Smirnov (K-S) tests are used on both binary and non-binary features; the mean values of numeric features are also compared to examine univariate fidelity. For multivariate inspections, pairwise feature correlations in the synthetic data are examined and compared with those in the original data. Additionally, unsupervised methods like a bag-of-words (BoW) representation are used to compare the synthetic dataset to the original dataset. To measure the separability of the synthetic dataset from the original, a silhouette coefficient and a random forest classifier are used. A silhouette coefficient is a multivariate, unsupervised metric that compares one cluster against another and quantifies the overlap of the real and synthetic datasets.” ([0030] of Beigi); and “As mentioned above, a silhouette coefficient is a multivariate, unsupervised metric that compares one cluster against another and quantifies the overlap of the real and synthetic datasets. The closer the coefficient is to zero the better. The silhouette coefficient for the inventive method is -0.001, which is much better than that of the other three methods” ([0036], Table 1 of Beigi)).
Therefore, it would have been obvious to one of ordinary skill in the art at the effective filing date of the present application to modify the invention of Hameed in view of Beigi to include techniques of generating a pre-ML score based on a statistical comparison of the empirically collected dataset and the synthetic dataset. Doing so would result in an improved invention because this would allow the system to evaluate how well the source’s underlying properties are preserved.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hameed (“Improving mortality prediction in Acute Pancreatitis by machine learning and data augmentation”, M. Asad Bin Hameed, Computers in Biology and Medicine 150 (2022) 106077, Sep 11, 2022), in view of Agarwal (EP 4141807 A1).
Regarding claim(s) 10, Hameed does not disclose, but Agarwal teaches:
generating a final score by averaging the Auto-ML score and the pre-ML score. (By disclosing, “the training model for obtaining the generator model may be selected from a plurality of given training models depending on one or more scoring metrics, particularly an average scoring metrics of multiple scoring metrics.” ([0009] of Agarwal); and “As a statistical metrics as the scoring metrics a score of the similarity between the distributions of the real training data and the synthetic training data is made. This can be made using a Kolmogorov-Smirnov test (KStest) wherein the distance between the empirical distribution functions is quantified. It is based on the null hypothesis that the two samples are drawn from the same continuous distribution and an alternate hypothesis that the two samples are drawn from two different distributions with 95% critical value. In this framework, 0 and 1 are the statistical minima and maxima of the average score across all the corresponding features in the real and synthetic data respectively. The higher the score, the higher is the quality of the generated synthetic data.” ([0056]-[0057] of Agarwal)).
Therefore, it would have been obvious to one of ordinary skill in the art at the effective filing date of the present application to modify the invention of Hameed in view of Beigi to include techniques of generating a final score by averaging the Auto-ML score and the pre-ML score. Doing so would result in an improved invention because this would allow the system to provide a better quantitative understanding of the scores.
Allowable Subject Matter
Claim(S) 8 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 101 set forth in this Office action.
As per claim 8, the closest prior art of record, Hameed (“Improving mortality prediction in Acute Pancreatitis by machine learning and data augmentation”, M. Asad Bin Hameed, Computers in Biology and Medicine 150 (2022) 106077, Sep 11, 2022) discloses an invention for obtaining an empirically collected dataset; generating a synthetic dataset based on the empirically collected dataset; performing an automated machine learning (Auto-ML) analysis of the synthetic dataset; and generating an Auto-ML score of the synthetic dataset based on a result of the Auto-ML analysis. In addition, Soni (US 20200134446) discloses an invention for training a synthetic data machine learning system with a training portion of the synthetic dataset; training a real data machine learning system with a training portion of the empirically collected dataset; inputting a testing portion of the synthetic dataset to the trained synthetic data machine learning system; inputting a testing portion of the empirically collected dataset to the trained real data machine learning system; and inputting the testing portion of the empirically collected dataset to the trained synthetic data machine learning system, wherein the Auto-ML score is based on a comparison of the outputs of the trained real data and synthetic data machine learning systems.
The closest prior art of record fail to teach or suggest, in the context of the ordered combination of the claim 8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
EP 3640861 A1 to Truong for disclosing:
systems and methods for parsing unstructured data with neural networks. In one implementation, a system for parsing unstructured data may include at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system to: receive unstructured data; apply a classifier to the unstructured data to identify a type of the unstructured data; based on the identification, select a corresponding neural network; apply the selected neural network to the unstructured data to obtain structured data; and output the structured data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUAN ZHANG whose telephone number is (571)272-4642. The examiner can normally be reached Mon - Fri 10 AM-5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Neha Patel can be reached at 571-270-1492. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUAN ZHANG/Primary Examiner, Art Unit 3699
Prosecution Insights