NON-FINAL REJECTION, FIRST DETAILED ACTION
Status of Prosecution
The present application 18/195,115, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The application was filed in the Office on May 9, 2023. There is a related case, PCT/US24/28080, filed May 7, 2024.
Claims 1-27 are pending and all are rejected in this rejection. Claims 1, 15 and 27 are independent claims.
Status of Claims
Claims 1-5, 8-19 and 22-27 are rejected under 35 USC § 103 as being unpatentable by Zhang et al. (“Zhang”), United States Patent Application Publication 2019/0370955 published on Dec. 5, 2019 in view of non-patent literature Lakshminarayanan et al. (“Lakshminarayanan”), “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, published in 2017.
Claims 6-7 and 20-21 are rejected under 35 USC § 103 as being unpatentable by Zhang in view Lakshminarayanan in further view of Cheng et al. (“Cheng”), United States Patent Application Publication 2021/0357402 published on Nov. 18, 2021.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
A.
Claims 1-5, 8-19 and 22-27 are rejected under 35 USC § 103 as being unpatentable by Zhang et al. (“Zhang”), United States Patent Application Publication 2019/0370955 published on Dec. 5, 2019 in view of non-patent literature Lakshminarayanan et al. (“Lakshminarayanan”), “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, published in 2017.
As to Claim 1, Zhang teaches: A characterization system, the characterization system comprising:
one or more controllers including one or more processors configured to execute a set of program instructions stored in memory (Zhang: par. 0043, processors and memory medium), the set of program instructions configured to cause the one or more processors to:
train a plurality of machine learning models based on a set of training data, the set of training data including empirical data labeled based on known information or simulated data labeled based on known information (Zhang: par. 0083, multiple models are trained; pars. 0059, training multiple predictive models using datasets that include some labeled data);
a first machine learning model of the plurality of machine learning models being different from one or more additional machine learning models based on at least one of a set of hyperparameters or a dataset (Zhang: Fig,. 3, par. 0083, there are multiple models that are trained based on datasets);
receive a plurality of sample measurement datasets from one or more test samples (Zhang: pars. 0058-59, the data points that are input to acquisition [202] may be from one or more specimens (i.e. test samples));
for each of the plurality of sample measurement datasets:
apply each trained machine learning model to determine a measurement value (Zhang: par. 0085, the computer may apply the models on the sub-pool of data); and
generate a measurement output based on N trained machine learning models, wherein the N trained machine learning models are a sub-set of the plurality of trained machine learning models (Zhang: par. 0090, Fig. 3, the acquisition score [322] (i.e. measurement output) is generated)).
PNG
media_image1.png
790
621
media_image1.png
Greyscale
Zhang may not explicitly teach: train a plurality of machine learning models based on a set of training data, the set of training data including empirical data labeled based on known information or simulated data labeled based on known information, each machine learning model of the plurality of machine learning models capable of generating an uncertainty estimator;
apply each trained machine learning model to determine a measurement value and the uncertainty estimator for each trained machine learning model; and
generate a measurement output based on N trained machine learning models with the lowest uncertainty estimators, wherein the N trained machine learning models are a sub-set of the plurality of trained machine learning models.
Zhang does further teach that from limited data points, the trained models may allow the prediction to generate an uncertainty estimation (Zhang: par. 0083). Lakshminarayanan teaches in general concepts related to quantifying predictive uncertainty in neural networks (Lakshminarayanan: Abstract). Lakshminarayanan teaches that “predictive uncertainty can be estimated using an ensemble of neural networks” and that uncertainty is derived from variability among model outputs (Lakshminarayanan: Section 2.1). Scoring rules assign numerical scores to predictive distributions, rewarding better calibrated predictions over worse (Lakshminarayanan: Section 2.2). Examiner asserts that this scoring scheme is used to determine the best model, which would be based on the best (or lowest uncertainty score) distributions.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Zhang disclosures and teachings by generating output with the lowest uncertainty estimators as taught and suggested by Lakshminarayanan. Such a person would have been motivated to do so with a reasonable expectation of success to ensure optimization of the machine learning systems with minimized uncertainty.
As to Claim 2, Zhang and Lakshminarayanan teach the limits of claim 1.
Zhang further teaches: wherein N is an integer equal to one (Zhang: par. 0083, the number may be a range 1..K, which Examiner asserts could be just a subset of 1).
As to Claim 3, Zhang and Lakshminarayanan teach the limits of claim 2.
Zhang and Lakshminarayanan as combined further teaches: wherein the generate a measurement output based on N trained machine learning models with the lowest uncertainty estimator, wherein the N trained machine learning models are a sub-set of the plurality of trained machine learning models comprises:
selecting one trained machine learning model with the lowest uncertainty estimator from the plurality of trained machine learning models; and
provide an associated measurement value of the selected trained machine learning model with the lowest uncertainty estimator as the measurement output (Lakshminarayanan: Section 2.2, scoring; Examiner asserts that this scoring scheme is used to determine the best model, which would be based on the best (or lowest uncertainty score) distributions).
As to Claim 4, Zhang and Lakshminarayanan teach the limits of claim 1.
Zhang further teaches: wherein N is an integer equal to two or greater than two (Zhang: par. 0083, the number may be a range 1..K, which Examiner asserts could be at least two).
As to Claim 5, Zhang and Lakshminarayanan teach the limits of claim 4.
Zhang and Lakshminarayanan as combined further teaches: wherein the generate a measurement output based on N trained machine learning models with the lowest uncertainty estimator, wherein the N trained machine learning models are a sub-set of the plurality of trained machine learning models comprises:
select two or more trained machine learning models with the lowest uncertainty estimators from the plurality of trained machine learning models; and
generate the measurement output by averaging the associated measurement values of the selected two or more trained machine learning models with the lowest uncertainty estimators (Lakshminarayanan: Section 2.2, scoring; Examiner asserts that this scoring scheme is used to determine the best model, which would be based on the best (or lowest uncertainty score) distributions).
As to Claim 8, Zhang and Lakshminarayanan teach the limits of claim 1.
Lakshminarayanan further teaches: wherein the set of hyperparameters comprise at least one of: neural network layers, neurons, regularization, dropout layers, Monte Carlo dropout (Lakshminarayanan: sec. 1, citing to Gal et al. for using Monte Carlo dropout, or Bayesian neural networks.
As to Claim 9, Zhang and Lakshminarayanan teach the limits of claim 1.
Zhang further teaches: wherein the plurality of machine learning models comprise at least one of:
a deep learning regression model (Zhang: par. 0071), an ensemble learning algorithm (Zhang: par. 0073), an artificial neural network, a convolutional neural network, or a residual neural network.
As to Claim 10, Zhang and Lakshminarayanan teach the limits of claim 1.
Lakshminarayanan further teaches: wherein the uncertainty estimator includes at least one of: Bayesian Neural Networks, Monte Carlo Dropout (Lakshminarayanan: sec. 1, citing to Gal et al. for using Monte Carlo dropout), or Deep ensembles.
As to Claim 11, Zhang and Lakshminarayanan teach the limits of claim 1.
Zhang further teaches: a metrology sub-system communicatively coupled to the one or more controllers (Zhang: par. 0056, metrology applications).
As to Claim 12, Zhang and Lakshminarayanan teach the limits of claim 11.
Zhang further teaches: wherein the metrology sub-system comprises at least one of: a spectroscopic ellipsometer, a reflectometer, a small angle x-ray scatterometer, a scanning electron microscope, a transmission electron microscope, or an optical sub-system (Zhang: par. 0029, the imaging subsystem can be one of different things including optical light scanning).
As to Claim 13, Zhang and Lakshminarayanan teach the limits of claim 1.
Zhang further teaches: wherein the sample comprises a substrate (Zhang: par. 0003).
As to Claim 14, Zhang and Lakshminarayanan teach the limits of claim 13 (Zhang: par. 0003).
Zhang further teaches: wherein the substrate comprises a wafer.
As to Claim 15, it is rejected for similar reasons as claim 1. Examiner asserts that a characterization subsystem that is coupled to controllers to perform similar functions as those recited in claim 1 is operatively similar for rejection purposes.
As to Claim 16, it is rejected for similar reasons as claim 2.
As to Claim 17, it is rejected for similar reasons as claim 3.
As to Claim 18, it is rejected for similar reasons as claim 4.
As to Claim 19, it is rejected for similar reasons as claim 5.
As to Claim 22, it is rejected for similar reasons as claim 8.
As to Claim 23, it is rejected for similar reasons as claim 9.
As to Claim 24, it is rejected for similar reasons as claim 10.
As to Claim 25, it is rejected for similar reasons as claim 11.
As to Claim 26, it is rejected for similar reasons as claim 12.
As to Claim 27, it is rejected for similar reasons as claim 1.
B.
Claims 6-7 and 20-21 are rejected under 35 USC § 103 as being unpatentable by Zhang et al. (“Zhang”), United States Patent Application Publication 2019/0370955 published on Dec. 5, 2019 in view of non-patent literature Lakshminarayanan et al. (“Lakshminarayanan”), “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, published in 2017 in further view of Cheng et al. (“Cheng”), United States Patent Application Publication 2021/0357402 published on Nov. 18, 2021.
As to Claim 6, Zhang and Lakshminarayanan teach the limits of claim 1.
Zhang and Lakshminarayanan may not explicitly teach: wherein the first machine learning model includes a first set of hyperparameters and the one or more additional machine learning models include one or more additional sets of hyperparameters, where the one or more additional sets of hyperparameters of the one or more additional machine learning models are different from the first set of hyperparameters of the first machine learning model.
Cheng teaches in general concepts related to time series forecasting that determines which model a plurality of models best first the forecast (Cheng: Abstract). Specifically, Cheng teaches hyperparameters for the different models may be tuned for the training (Cheng: par. 0025, hyperparameter tuning may be performed). Each of the models may be trained with different parameters (Cheng: par. 0024).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Zhang- Lakshminarayanan disclosures and teachings by implementing the methods and systems with hyperparameters for the different models as taught and disclosed by Cheng. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the different models to be separately adapted.
As to Claim 7, Zhang and Lakshminarayanan teach the limits of claim 1.
Zhang and Lakshminarayanan may not explicitly teach: where the first machine learning model includes a first dataset and the one or more additional machine learning models include one or more additional datasets, where the one or more additional datasets of the one or more additional machine learning models are different from the first dataset of the first machine learning model.
Cheng teaches in general concepts related to time series forecasting that determines which model a plurality of models best first the forecast (Cheng: Abstract). Specifically, Cheng teaches hyperparameters for the different models may be tuned for the training (Cheng: par. 0025, hyperparameter tuning may be performed). Each of the models may be trained with different parameters (Cheng: par. 0024). Datasets may be current ones, which may be updated over time (Cheng: par. 0023). Examiner asserts that this would be different data sets and similar to the hyperparameters, have different ones for different models.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Zhang- Lakshminarayanan disclosures and teachings by implementing the methods and systems with datasets for the different models as taught and disclosed by Cheng. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the different models to be separately adapted.
As to Claim 20, it is rejected for similar reasons as claim 6.
As to Claim 21, it is rejected for similar reasons as claim 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES T TSAI whose telephone number is (571)270-3916. The examiner can normally be reached M-F 8-5 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached on 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES T TSAI/Primary Examiner, Art Unit 2174
Prosecution Insights