DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in responsive to communication(s): original application filed on 02/06/2024. Claims 1-16 remain pending. Claims 1 and 9 are independent.
Double Patenting
Claims 1-16 of this application is patentably indistinct from Claims 1-16 of Application No. 18/615,188. Pursuant to 37 CFR 1.78(f), when two or more applications filed by the same applicant or assignee contain patentably indistinct claims, elimination of such claims from all but one application may be required in the absence of good and sufficient reason for their retention during pendency in more than one application. Applicant is required to either cancel the patentably indistinct claims from all but one application or maintain a clear line of demarcation between the applications. See MPEP § 822.
A rejection based on double patenting of the “same invention” type finds its support in the language of 35 U.S.C. 101 which states that “whoever invents or discovers any new and useful process... may obtain a patent therefor...” (Emphasis added). Thus, the term “same invention,” in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957).
A statutory type (35 U.S.C. 101) double patenting rejection can be overcome by canceling or amending the claims that are directed to the same invention so they are no longer coextensive in scope. The filing of a terminal disclaimer cannot overcome a double patenting rejection based upon 35 U.S.C. 101.
Claims 1-16 are provisionally rejected under 35 U.S.C. 101 as claiming the same invention as that of Claims 1-16 of co-pending Application No. 18/615,188 (reference application). This is a provisional statutory double patenting rejection since the claims directed to the same invention have not in fact been patented.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because (1) reference character “250” has been used to designate both "Generate a model of best fit based on the data" in FIG. 2 and "... generates a report" in ¶ [0040]; and (2) reference character “260” has been used to designate both "Generate a report" in FIG. 2 and "... generates a future test point based on the validated model" in ¶ [0041]. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because (1) reference characters "250" in ¶ [0040] and "260" in FIG. 2 have both been used to designate "Generate a report"; and (2) reference characters "260" in ¶ [0041] and "280" in FIG. 2 have both been used to designate "Generate a future training/test point based on the validation/validated model"; and (3) reference characters "320" in ¶¶ [0043]-[0044], "350" in ¶¶ [0044]-[0045], and "380" in ¶ [0046] have all been used to designate "Framework" Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: (1) . Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to because reference character(s) used in FIG. 3 is inconsistent and confused with the description in ¶¶ [0043]-[0046] of the Specification. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
the description in ¶¶ [0043]-[0046] is inconsistent and confused with reference character(s) used in FIG. 3 (e.g., "320", "350", "380" are all referred to "Framework" in ¶¶ [0043]-[0046] but shown as different elements in FIG. 3, see also Drawings Objections).
Appropriate correction is required.
Claim Objections
Claims 1, 5, 7, 9, 13, and 15 are objected to because of the following informalities:
in Claim 1, lines 6-9; and Claim 9, lines 6-9, "… determining/determine, by the uncertainty quantification computing device, whether the input parameters fall within an accuracy range; generating/generate, by the uncertainty quantification computing device, a report including the plurality of the uncertainty intervals and a future test point recommendation." appears to be "… determining/determine, by the uncertainty quantification computing device, whether the input parameters fall within an accuracy range; and generating/generate, by the uncertainty quantification computing device, a report including the plurality of the uncertainty intervals and a future test point recommendation.";
in Claim 5, line 6; and Claim 13, line 6, "… processing/process assumptions using hyper-parameter optimization before performing the statistical analysis on the input parameters; utilizing/utilize a statistical test to analyze the performance of the surrogate models" appears to be "… processing/process assumptions using hyper-parameter optimization before performing the statistical analysis on the input parameters; and utilizing/utilize a statistical test to analyze performance of the surrogate models";
in Claim 7, lines 1-2; and Claim 15, lines 1-2, "… wherein generating the report including the plurality of uncertainty intervals and future test point recommendations comprises …" appears to be "… wherein generating the report including the plurality of uncertainty intervals and the future test point recommendations comprises …"; .
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 11 recites the limitation "... wherein in performing the statistical analysis to generate " in lines 1-2, which rendering the claim indefinite because ".
Claims 12-14 are rejected for fully incorporating the deficiency of their respective base claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefore, subject to the conditions and requirements of this title.
Claims 1-16 are rejected under 35 U.S.C. 101 because
the claimed invention is directed to abstract idea without significantly more.
Independent Claims 1 and 9
Step 1: Claim 1 is a process claim and Claim 9 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) recite(s) ".
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) recite(s) additional elements/limitations of ".
Step 2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional limitation/element of .
Claims 2 and 10
Step 1: Claim 2 is a process claim and Claim 10 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) further recite(s) ".
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) further recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional limitation/element of ".
Claims 3 and 11
Step 1: Claim 3 is a process claim and Claim 11 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) further recite(s) ".
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) further recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional limitation/element of ".
Claims 4 and 12
Step 1: Claim 4 is a process claim and Claim 12 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) further recite(s) ".
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 5 and 13
Step 1: Claim 5 is a process claim and Claim 13 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) further recite(s) ".
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 6 and 14
Step 1: Claim 6 is a process claim and Claim 14 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) further recite(s) ".
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) does/do not further recite(s) additional elements/limitations.
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception. Thus, none of the additional limitations, taken either alone or combined, amount to significantly more than the abstract idea.
Claims 7 and 15
Step 1: Claim 7 is a process claim and Claim 15 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) does/do not further recite(s) elements/limitations which can be reasonably considered as mental processes (i.e., which "can be performed in the human mind, or by a human using a pen and paper") or mathematical concepts/algorithms/calculations.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) further recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional limitation/element of ".
Claims 8 and 16
Step 1: Claim 8 is a process claim and Claim 16 is a system claim. These claims fall within at least one of the four categories of patent eligible subject matter.
Step 2A Prong 1: The claim(s) further recite(s) ".
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the claim(s) further recite(s) additional element/limitation of ".
Step 2B: The claim(s) does/do not further include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional limitation/element of ".
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-13, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over RAMANATH et al. (WO 2024/199632 A1, filed on 03/28/2023), hereinafter RAMANATH in view of AHLFELD et al. (US 2024/0135259 A1, filed on 12/18/2023), hereinafter AHLFELD and KAR et al. (US 2025/0036835 A1, priority date: 07/17/2023), hereinafter KAR.
Independent Claims 1 and 9
RAMANATH discloses a method for quantifying uncertainty in computational based models (RAMANATH, Page 1, lines 4-24: the present invention relates to uncertainty quantification in mathematical models, and more particularly relates to a computer-implemented method and system for performing uncertainty quantification based on a user's expertise level; mathematical models are developed, in general, to model physical as well as behavioral characteristics of a system such as, but not limited to, engineering tools, machines etc.; the term 'mathematical models' as used herein may encompass physics-based models or hybrid models that are at least partially based on statistics; to use mathematical models effectively, uncertainties associated with outputs given the uncertainties in the inputs and vice-verse need to be quantified; uncertainty quantification is used to quantify risk on responses of a given system due to variability and uncertainty in input parameters and/or system or model parameters; Page 2, lines 9-18: manage uncertainty quantification in an engineering application; receiving a request comprising contextual data for performing an uncertainty quantification, from a user interface, wherein the contextual data is indicative of at least one of a scenario and one or more constraints associated with the scenario; Page 8, lines 10-29: perform uncertainty quantification based on a given scenario, using a corresponding mathematical model in a simulation environment; the term 'simulation environment' as used herein refers to a programming environment of a computer system that enables execution of system simulations based on requirements of a user; the simulation environment facilitates running of model-based experiments based on discrete event simulation, dynamic simulation and process simulations; the simulations may be associated with a product, process, a plant or an event; Page 9, lines 10-19: to enable performing of uncertainty quantification, enable the user to provide contextual information for a scenario corresponding to the uncertainty quantification in the form of natural language text, voice, sketches or interactive controls based on a level of expertise of the user; the term 'scenario' as used herein refers to a phenomenon that may be represented using a mathematical model), the method comprising:
receiving, by an uncertainty quantification computing device (RAMANATH, Page 9, line 24 – Page 10, line 13 with 102 and 106 FIG. 1: a simulation platform 102 including at least one processor 104 that is configured to execute at least one uncertainty quantification module 106 from a memory 108 accessed by the processor 104; simulation platform 102 may include functionalities of performing uncertainty quantification by, for example, estimating one or more variable distributions associated with one or more uncertain variables, executing an uncertainty quantification algorithm based on the estimated one or more variable distributions, and providing a user-interpretable report based on an output of the uncertainty quantification algorithm; the described uncertainty quantification module 106 may include and/or correspond to one or more components of the simulation platform 102 that is configured to perform uncertainty quantification), input parameters (RAMANATH, Page 7, lines 1-3 with FIG. 9: a Graphical User Interface showing input parameters and output variables for a two-tank process; Page 11, lines 3-15: the inputs may include a sketch associated with one or more uncertain variables corresponding to an associated scenario; the sketch may be indicative of a probability distribution (e.g., Gaussian distribution) indicative of a nature of an uncertain variable; the inputs may include a natural language text indicative of the scenario, the uncertain variables and associated constraints; Page 12, line 12 – Page 15, line 3 with FIG. 1: receive a request comprising contextual data for performing an uncertainty quantification, from the user interface 116 of the computer system 100; the contextual data is indicative of at least one of a scenario, and the one or more constraints associated with the scenario; the user may firstly select a scenario from a plurality of predefined scenarios; based on the scenario selected by the user, the uncertainty quantification module 106 identifies a predefined mathematical model corresponding to the scenario; the mathematical model may include, for example, statistical models, physics-based, probabilistic models or hybrid models; the one or more constraints may include upper limits, lower limits or ranges associated with one or more uncertain variables corresponding to the scenario; in case of a novice, enable the user to provide the contextual data in the form of one or more sketches, natural language text or voice input; the user may choose to provide a sketch-based input by using a digital pen that enables the user to draw sketches of variable distributions associated with the uncertain variables on the user-interface; in case of an expert user, enable the user to provide the contextual data by making selections using interactive controls on the user interface 116; the selections may correspond to constraints to be applied while identifying the appropriate uncertainty quantification algorithm; e.g., a first selection may be associated with selecting one of 'parameter estimation', 'sensitivity analysis' and 'forward uncertainty quantification'; if the first selection is 'parameter estimation', further selections may correspond to application of probabilistic inverse methods, Bayesian methods, etc.; Page 18, line 19 – Page 20, line 9 with FIGS. 2-3: the user selects a scenario corresponding to the uncertainty quantification via interactive controls; based on the scenario selected by the user, a series of questions are generated; an exemplary scenario of time of arrival at office from home (henceforth called 'Office Arrival Time' is considered here) is considered, where the user is interested in estimating uncertainty propagation in a mathematical model corresponding to the scenario; the mathematical model corresponding to 'Office Arrival Time' may estimate a time at which a person may reach his/her office, based on uncertain variables such as start time Tstart, vehicle type Vtype used for commute, and route choice Rchoice; the uncertain variables Vtype and Rchoice further enable determination of variables speed 's' and distance 'd', respectively; the statistical model corresponding to the scenario may be as shown in Eqn. (1); a Directed Acyclic Graph (DAG) representation 300 of the corresponding probabilistic model of the scenario is shown in FIG 3; the DAG 300 comprises nodes corresponding to each of the variables associated with the scenario, with nodes interconnected using directed edges that indicate dependency between the variables; Page 31, line 16 – Page 32, line 10 with FIGS. 8-9: a two-tank model 905 created in a process simulation environment acts as a process simulation model; the two-tank model 905 is associated with 14 input variables 910 and 17 output variables 915, as shown in Graphical User Interface 900; at step 802, a request for performing an uncertainty quantification is received in the form of responses to a series of prompts generated, by the computer system 100, on the user 30 interface 116; the responses to the series of prompts forms the contextual data associated with the uncertainty quantification; the computer system 100 prompts the user to enter number of uncertain variables associated with the scenario via the user interface 116; based on the number of uncertain variables, the user is provided an option to enter variable definitions using interactive control elements provided on the user interface 116);
performing, by the uncertainty quantification computing device, a statistical analysis to generate a plurality of uncertainty intervals based on the input parameters (RAMANATH, Page 2, line 26 – Page 3, line 22: estimating one or more variable distributions associated with one or more uncertain variables corresponding to the scenario, using the contextual data; the variable distribution is computed based on the one or more component functions and the one or more sufficient statistics; Page 9, lines 15-17: the contextual information is further used to estimate variable distributions corresponding to one or more uncertainty variables associated with the scenario; Page 11, line 26 – Page 12, line 10: the term 'sufficient statistics' as used herein is function of a sample data that contains relevant information about the uncertain variable; in other words, a sufficient statistic is a summary 30 of the data that retains all the relevant information about the parameter being estimated; sufficient statistics help us in reducing the dimensionality of data and in simplifying the analysis; by using a sufficient statistic instead of the full dataset, redundant calculations may be avoided, and the efficiency of statistical inference may be improved; Page 13, lines 20-28: the one or more uncertain variables associated with the scenario includes input random variables or output random variables; the term 'random variable' as used herein refers to a variable that may not be defined with certainty, i.e., a variable whose value at a specific instance is not known with a certain degree of confidence; uncertainty quantification algorithms may include parameter estimation, sensitivity analysis, forward uncertainty quantification; Page 15, line 5 – Page 16, line 31 with FIG. 1: estimate one or more variable distributions associated with one or more uncertain variables corresponding to the scenario, using the contextual data; the uncertainty quantification module 106 may parse the natural language text, for example, using a text parser script; further, a vector embedding corresponding to the one or more keywords is generated using a vectorization function; the first classification model is a function that may be trained to classify the vector embeddings into one of a predefined category of component functions associated with the one or more uncertain variables present in the contextual data; the output of the first classification model is a label indicative of one or more component functions of the variable distribution associated with the uncertain variable; the term 'component functions' as used herein refer to mathematical functions present in the variable distribution corresponding to an uncertain variable; upon determining the component functions, one or more sufficient statistics corresponding to the one or more keywords are identified from the data store 122, using a rule-based matching algorithm; the variable distribution associated with the uncertain variable is estimated based on the one or more component functions and the one or more sufficient statistics; if the contextual data is received in the form of sketch input, a second classification model may be applied to the one or more sub-figures; the output of the second classification model is a label indicative of one or more component functions of the variable distribution; Page 23, line 9 – Page 27, line 7 with FIG. 5: estimating variable distribution for an uncertain variable, using contextual data received from a user; at step 505, the text is preprocessed to identify subphrases within the text input; each of the subphrases may correspond to a different component function; the term 'component function' as used herein refers to a probability distribution function that forms part of the variable distribution being estimated; at step 510, a first classification model is applied to the vectorized keywords corresponding to each of the subphrases; at step 515, the first classification model generates an output indicative of the component function associated with the respective subphrase; at step 520, sufficient statistics associated with the component function corresponding to each of the subphrases are identified; at step 535, the variable distribution for the uncertain input variable is computed based on the component functions and the sufficient statistics; the variable distributions corresponding to uncertain input variables such as Vtype, Rchoice etc. in the probabilistic model may also be determined; a forward uncertainty quantification algorithm may be used to quantify uncertainty propagated through the probabilistic model based on the variable distributions associated with the uncertain input variables; the choice of forward uncertainty quantification algorithm may be determined using a third classification model; Page 27, line 29 – Page 28, line 27 with FIG.6: estimating variable distribution for an uncertain variable, based on a sketch input; at step 605, the sketch input is preprocessed to identify one or more sub-figures; at step 610, a second classification model is applied on each of the sub-figures in the sketch, to identify the one or more component functions; at step 615, the convolutional neural network may generate an output label indicative of the component function corresponding to the respective sub-figure; at step 620, one or more sufficient statistics are estimated for the component functions by using a rule-based matching algorithm; at step 635, the variable distribution for the uncertain variable is computed based on the component functions and the sufficient statistics; Page 32, line 10 – Page 33, line 20: each of the variable definitions include a variable name and choice of a variable distribution corresponding to the variable name, along with one or more sufficient statistics associated with the variable distribution; the one or more sufficient statistics may include, for example, mean and standard deviation for normal distribution, and an upper limit and a lower limit for uniform distribution; when the user is defining the variable definitions using the interactive controls, one or more recommendations for variable distributions may be dynamically generated based on the variable name entered by the user; Page 34, line 6 – Page 35, line with FIG. 10: if the user is unaware of parameter range and prior distributions corresponding to one or more input variables to a model, then parameter estimation or inverse uncertainty quantification is performed; at step 1002, contextual data comprising one or more samples generated via Latin Hypercube Sampling (LHS) technique are received; at step 1004, a Design Of Experiments (DoE) is conducted on the simulation platform 102 based on the LHS samples; at step 1006, if bounds of a variable leads to an error/divisible by zero/singularity, such bounds are changed using sensitivity or SOBOL index); determining, by the uncertainty quantification computing device, whether the input parameters fall within an expectation range (RAMANATH, Page 3, line 23 – Page 4, line 10: analyzing the contextual data to identify a user intent; the uncertainty quantification algorithm for the scenario is selected from among a plurality of uncertainty quantification algorithms, based on the user intent; the uncertainty quantification algorithm is associated with one of forward uncertainty quantification, reverse uncertainty quantification, and parameter sensitivity analysis; executing an uncertainty quantification algorithm based on the estimated variable distribution associated with each of the one or more uncertain variables and a mathematical model representative of the scenario, to generate an uncertainty quantification output; Page 17, lines 1-19: execute an uncertainty quantification algorithm based on the estimated variable distribution associated with each of the one or more uncertain variables 5 and the mathematical model representative of the scenario, to generate an uncertainty quantification output; analyze the contextual data to identify the user intent; the uncertainty quantification algorithm for the scenario is selected from among a plurality of uncertainty quantification algorithms based on the user intent; the user intent is indicative of the type of uncertainty quantification algorithm to be chosen for the scenario; the third classification model selects the uncertainty quantification algorithm based on the one or more constraints present in the contextual data; Page 27, lines 9-27: the text input from the user comprises 10 an expectation for an output variable; the text input may also comprise one or more prior beliefs associated with the uncertain input variables; the term 'prior belief' as used herein refers to information known to the user about the uncertain input variables; the text inputs corresponding to each of the expectations and prior beliefs may be provided in response to the questionnaire template; variable distributions associated with the output variable and the one or more uncertain input variables are estimated, from the text input, by following the steps 505 to 525; subsequently, a reverse uncertainty quantification algorithm may be applied to the variable distributions in order to generate an output indicative of parameter estimates associated with the probabilistic model, in order to achieve the expectation of the user; Page 28, line 29 – Page 31, line 14 with FIGS. 1 and 7A-B: at step 702, a user intent is identified by analyzing contextual data received via the user interface 116; the user intent indicates a nature of uncertainty quantification to be performed; the user intent may be identified based on response of the user to at least one of the questionnaire templates; the questionnaire template may be, e.g., "Accuracy and computational effort are more or less inversely proportional. More the computational effort, more is the cost in resources and time. Keeping this in mind, please provide a number on the scale of 5-10 that indicates the importance of the cost of computational efforts involved."; at step 704, an uncertainty quantification algorithm is selected from a plurality of uncertainty quantification algorithms by applying a third classification model to the user intent; Page 33, line 12 – Page 34, line 4: upon definition of the variables, a series of prompts are generated to identify the user intent; the prompts may be generated using questionnaire templates; some examples of prompts include "Do you have a faster Analysis Driver?", "Do you want to use a metamodel or a surrogate model?", "Which physics domain are you working on (e.g. Hydraulics/ Pneumatics, Electrical, Controls, Thermodynamics, Mechanical or Chemical)?", "What is the 20 intended study?", "Do you want to understand the variance/spread in the output response?" etc.; the uncertainty quantification output comprises one or more variable distributions corresponding to one or more of the output variables); and
generating, by the uncertainty quantification computing device, a report including the plurality of the uncertainty intervals (RAMANATH, Page 4, lines 12-20: generating a report based on the uncertainty quantification output; deriving one or more key parameters from the uncertainty quantification output based on a non-parametric testing algorithm; the generated report is further provided on the user interface; Page 9, lines 20-23: facilitate generation of user-interpretable reports based on an uncertainty quantification output generated via the uncertainty quantification performed based on the estimated variable distributions; Page 11, lines 15-21 with FIG. 1: the graphical user interface 116 may include an output user interface (UI) 118 provided to the user for viewing one or more outputs associated with the uncertainty quantification; the output UI 118 further includes a variable distribution UI 120 that enables the user to view variable distributions associated with the uncertain variables; Page 17, line 21 – Page 18, line 7 with FIG. 1: generate a report based on the generated uncertainty quantification output; the report comprises a graphical representation of the uncertainty quantification output; generating the report based on the uncertainty quantification output comprises deriving one or more key parameters from the uncertainty quantification output based on a non-parametric testing algorithm; the report is generated based on a template; provide the generated report on the user interface 116; the report may be generated in the form of graphical representations in case of expert users, and in the form of natural language text for novices; Page 36, line 7 – Page 38, line 15 with FIGS. 12 and 13A-C: generating a user-interpretable report based on an outcome of performing an uncertainty quantification; the user-interpretable report is a summarized form of the variable distributions associated with the uncertain variables determined through the uncertainty quantification; at step 1202, variable distributions estimated through the uncertainty quantification is subjected to a non-parametric test such as Kolmogorov-Smirnov test; at step 1204, one or more key parameters associated with the identified variable distribution is derived using a predefined library, for example, SciPy; at step 1206, the one or more key parameters are embedded into a natural language text using a natural language generation (NLG) technique).
RAMANATH further discloses a computer system for quantifying uncertainty (RAMANATH, Page 9, lines 24-28 with 100 in FIG. 1: a computer system or data processing system 100 for performing of uncertainty quantification based on expertise level of a user), said system comprising: a computing device (RAMANATH, Page 9, line 29 with 102 in FIG. 1: the computer system 100 may include a simulation platform 102) comprising a processor (RAMANATH, Page 9, lines 29-32 with 104 in FIG. 1: the computer system 100 may include a simulation platform 102 including at least one processor 104 that is configured to execute at least one uncertainty quantification module 106 from a memory 108 accessed by the processor 104), said processor configured to perform the method described above (RAMANATH, Page 9, line 32 – Page 10, line 23 with 100 in FIG. 1: simulation platform 102 may include functionalities of performing uncertainty quantification by, for example, estimating one or more variable distributions associated with one or more uncertain variables, executing an uncertainty quantification algorithm based on the estimated one or more variable distributions, and providing a user-interpretable report based on an output of the uncertainty quantification algorithm; the user may be located close to the simulation platform 102 or remote to the simulation platform 102, e.g., using a workstation for connecting to the simulation platform 102, e.g., via the internet, wherein the workstation may include an input device 112 and a display device 114).
RAMANATH fails to explicitly disclose (1) determining whether the input parameters fall within an accuracy range; and (2) generating a report including a future test point recommendation.
AHLFELD teaches a system and a method relating to estimating uncertainty in complex engineering problems (AHLFELD, ¶¶ [0009]-[0010], [0207], [0214], and [0217] with FIG. 22), wherein determining whether the input parameters fall within an accuracy range (AHLFELD, ¶ [0217] with FIG. 22: FIG. 22 shows the results of the scalar prediction using the neural network model previously applied, in which the predictions of the outputs are instantly displayed for a new selection of input parameters; sliders are provided to change any of the input parameters; the range of the input sliders is defined by the min and max values of the inputs in the training set of the model; each time an input value is modified, the output value changes accordingly; as the model used for the prediction was trained with the option to predict uncertainty, a shaded range around the dial that indicates the uncertainty of the prediction is provided; this is because some predictions are more uncertain than others, and this can help the end-user when making decisions; hence, the performance of a new design tested under new conditions can instantly be predicted, rather than waiting for hours of re-design, simulation, post processing to complete; ¶¶ [0247]-[0257] with FIGS. 40-44: once the model is trained, its quality/accuracy can be evaluated using the test data put aside as part of preprocessing; the end-user then selects the predicted vs. actual step of the model manipulator; test data and DragCoefficient is selected as the output; this may be used to evaluate how a model is predicting unseen data. In this case, the model has high accuracy, as the points are closed to the perfect prediction line as shown in FIG. 40; another way to evaluate the model is to import data which is especially designed to validate predictions; in this example, the validation data is a design sweep: AngleOfAttack is the only parameter changing between each simulation; using 'Model→Validation Plot', the validation data is selected for the test data, AngleOfAttack for the X-axis and LiftCoefficient for the yAxis; FIG. 41 provides the results which shows that the predictions are accurate; an important outcome of training the model is uncertainty quantification: there's a 95% probability that measured data will be contained within the uncertainty region; this helps an end-user knowing when the predictions can be trusted; advantageously, the platform provides an understanding of how data volumes affect the accuracy of machine learning models; in turns, this helps accelerate engineering development; additionally, an end-user may also want to understand the accuracy required to make a particular model useful; this may largely depend on a specific use case or downstream application; the platform may therefore predict the performance of a 3D object across multiple areas based on a specific downstream application; e.g., at the concept stage of engineering, a model with relatively low accuracy might suffice; when working towards the production/manufacturing stage of engineering and the end-goal is optimization, higher accuracy will be required; the end-user then selects 'Curve Prediction' as shown in FIG. 43; AngleOfAttack is selected as the input and DragCoefficient as the output; New design variables may be chosen or selected with sliders to make an instant assessment of spoiler performance; each time an input value is modified, the curve value changes accordingly; the results suggest that the uncertainty will be higher for higher values of AngleOfAttack; this makes physical sense since the flow around the spoiler will become more and more turbulent (and therefore harder to predict) as the angle of attack increases; as well as giving predictions of performance for a new selection of designs, the platform is also capable of providing optimal designs for a selection of target variables or performance metrics; the end-user then selects 'Apply→Min/Max Optimization' as shown in FIG. 44; this algorithm will use the Neural Network model to search for the best Lift-to-Drag ratio and will return the set of design parameters corresponding to the most efficient spoiler wing; this tool may be used for quickly identifying the best possible combination of design parameters which satisfy a target variable or performance); and (2) generating a report including a future test (AHLFELD, ¶¶ [0333]-[0341] with FIGS. 84-86: key parameters needed to enable a good model performance may be identified; identify key design parameters and to understand how to use these key parameters to improve the performance of the model; as shown in FIG. 84, the dataset is explored using the intelligent correlation step. Input_1, Input_3 and Input_ 4 show a correlation of circa 40% to the output variable, whilst the other inputs do not show a correlation. This is an early indication that these parameters might be beneficial to include as Inputs to the model whilst the others might not provide helpful information for the model to learn from; to understand which of the design parameters is having the biggest influence on the output, a sensitivity analysis step is performed; this step identifies the strength of the response of the output for each input; FIG. 86 shows in a visual way how much impact each input of a model has on the different outputs of this model; this plot indicates that Input_1, Input_2 and Input_3 are having the biggest influence on the output; this helps an end-user understand efforts should be focused on recording data about these inputs in future tests or simulations and still get a good prediction of a desired output; the step 'Apply-;, Explain predictions' is now used to understand the relative importance of input parameters in affecting design performance by quantifying the relative 'impact' of input parameters; this step helps build confidence in models by for example confirming engineering intuitions and get an understanding of where to concentrate the design/engineering efforts to improve product performance).
RAMANATH and AHLFELD are analogous art because they are from the same field of endeavor, a system and a method relating to estimating uncertainty in complex engineering problems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of AHLFELD to RAMANATH. Motivation for doing so would (1).
RAMANATH in view of AHLFELD fails to explicitly disclose generating a report including a future test point recommendation.
KAR teaches a system and a method relating to calculating uncertainty for producing an outcome within the desired range of the quality metrics specified at the outset (KAR, ¶¶ [0005] and [0042]-[0061]), wherein generating a report including a future test point recommendation (KAR, ¶¶ [0023]-[0061]: optimizing a fabrication process; receiving process data, wherein the process data includes a set of data points, each data point representing a set of one or more process parameters for the fabrication process and corresponding results for the fabrication process, wherein the set of data points are sampled from a defined design space of process parameter values, and wherein the results include a success indicator representing a successful fabrication and a quality metric representing a measurement of the fabrication process for each set of process parameters; selecting one or more recommended data points, each recommended data point representing a recommended set of process parameters, based on a ranking of the data points in the feasible region corresponding to the score of each data point; determining a set of one or more optimal data points from the recommended data points based on the corresponding quality metric satisfying a predetermined range or threshold for the quality metric; selecting one or more initial recommended data points based on a ranking of the data points in the initial feasible region corresponding to the initial score of each data point; repeating the method if the quality metric of the initial experimental results fails to satisfy the pre-determined range or threshold for the quality metric; selecting one or more data points from the defined design space, as one or more uncertain data points, based on a ranking of the data points of the defined design space corresponding to the uncertainty value of each data point; adding the one or more uncertain data points to the initial recommended data points; and wherein the one or more uncertain data points are included in the initial recommended data points for the performance of the fabrication process; determining the feasible region based on data points of the defined design space with corresponding probability value that exceeds a feasible threshold value; after selecting one or more recommended data points; sending instructions to a fabrication device, wherein the instructions include the one or more recommended data points and a command to execute one or more fabrication processes using the one or more recommended data points; ¶ [0129] with FIG.5: a query generator 425 was designed via multiple sampling strategies to generate the suggested DoE parameters for the next iteration of experiments (e.g., the 5 sets of tests described above); ¶ [0125] with FIG. 2: at step 255, the samples, or new parameters, from the sampling strategies are collected; the new parameters may then be provided for conducting experiments (e.g., the fabrication process) using those new parameters); the new parameters may be output, such as displayed on a screen, transmitted by a communication (e.g., email), or stored as data (e.g., a document or file) in a storage device; at step 260, experiments may be conducted using the new parameters; a user may access the new parameter values, such as viewing on a screen or opening a document, to perform the experiments).
RAMANATH in view of AHLFELD, and KAR are analogous art because they are from the same field of endeavor, a system and a method relating to calculating uncertainty for producing an outcome within the desired range of the quality metrics specified at the outset. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of KAR to RAMANATH in view of AHLFELD. Motivation for doing so would provide systematically improved conditions for higher-quality growth, and affirm the robustness of the constraint model's accuracy (KAR, ¶ [0129]).
Claims 2 and 10
RAMANATH in view of AHLFELD and KAY discloses all the elements as stated in Claims 1 and 9 respectively and further discloses wherein receiving the input parameters further comprises: defining simulation output parameters and the accuracy range; and uploading simulation output data and a configuration file (RAMANATH, Page 13, lines 18-28: the one or more constraints may include upper limits, lower limits or ranges associated with one or more uncertain variables corresponding to the scenario; the one or more uncertain variables associated with the scenario includes input random variables or output random variables. The term 'random variable' as used herein refers to a variable that may not be defined with certainty, i.e., a variable whose value at a specific instance is not known with a certain degree of confidence; Page 27, lines 9-27: the text input from the user comprises an expectation for an output variable; the text input may also comprise one or more prior beliefs associated with the uncertain input variables; the term 'prior belief' as used herein refers to information known to the user about the uncertain input variables; the text inputs corresponding to each of the expectations and prior beliefs may be provided in response to the questionnaire template; a reverse uncertainty quantification algorithm may be applied to 25 the variable distributions in order to generate an output indicative of parameter estimates associated with the probabilistic model, in order to achieve the expectation of the user; Page 28, line 33 – Page 29, line 17 with FIG. 7A: at step 702, a user intent is identified by analyzing contextual data received via the user interface 116; the user intent indicates a nature of uncertainty quantification to be performed; e.g., the user intent may be identified based on response of the user to at least one of the questionnaire templates; the questionnaire template may be, e.g., "Accuracy and computational effort are more or less inversely proportional. More the computational effort, more is the cost in resources and time. Keeping this in mind, please provide a number on the scale of 5-10 that indicates the importance of the cost of computational efforts involved."; in response, the user may provide a numerical input, for example, the number '8'. In another example, the user may provide a natural language text as input; e.g., the questionnaire template may be, "Do you have any limitations on number of iterations that may be performed during the uncertainty quantification?". The user may respond by typing 'Yes' or 'No'; the response of the user to the questionnaire template may be identified as the user intent; Page 31, line 16 – Page 33, line 10 with FIGS. 8-9: a two-tank model 905 created in a process simulation environment acts as a process simulation model; the two-tank model 905 is associated with 14 input variables 910 and 17 output variables 915, as shown in Graphical User Interface 900; at step 802, a request for performing an uncertainty quantification is received in the form of responses to a series of prompts generated, by the computer system 100, on the user 30 interface 116; the responses to the series of prompts forms the contextual data associated with the uncertainty quantification; the computer system 100 prompts the user to enter number of uncertain variables associated with the scenario via the user interface 116; based on the number of uncertain variables, the user is provided an option to enter variable definitions using interactive control elements provided on the user interface 116; each of the variable definitions include a variable name and choice of a variable distribution corresponding to the variable name, along with one or more sufficient statistics associated with the variable distribution; the one or more sufficient statistics may include, for example, mean and standard deviation for normal distribution, and an upper limit and a lower limit for uniform distribution; when the user is defining the variable definitions using the interactive controls, one or more recommendations for variable distributions may be dynamically generated based on the variable name entered by the user; upon definition of the variables, a series of prompts are generated to identify the user intent; the prompts may be generated using questionnaire templates; Page 34, line 6 – Page 34, line 12- with FIGS. 10-11: if the user is unaware of parameter range and prior distributions corresponding to one or more input variables to a model, then parameter estimation or inverse uncertainty quantification is performed; at step 1002, contextual data comprising one or more samples (henceforth called LHS samples) generated via Latin Hypercube Sampling (LHS) technique are received via the user interface 116; at step 1004, a Design Of Experiments (DoE) is conducted on the simulation platform 102 based on the LHS samples; the DoE is performed based on Monte Carlo or quasi-Monte Carlo analysis based on the LHS samples; at step 1006, if bounds of a variable leads to an error/divisible by zero/ singularity, such bounds are changed using sensitivity or SOBOL index; at step 1102, a set of Design of Experiments (DoE) data is received from the user; the DoE data may include one or more of a steady state configuration file, steady state measurement data, dynamic state configuration file and dynamic state measurement data obtained at step 1006 of method 1000) (AHLFELD , ¶¶ [0009]-[0010]: depending on the complexity of the problem and accuracy required, 3D simulations may be costly to compute; surrogate models may then be used for evaluating computationally expensive problems; a surrogate model is a simple model that approximate complex models using training samples; ¶ [0216] with FIG. 20: the end-user then chooses 'DragCoefficient' and 'LiftCoefficient' for the output; ¶ [0221] with FIG. 24: each CSV file imported contains one row which corresponds to the results of one CFD simulation; ¶ [0226] with FIG. 30: the simulation outputs are selected for the Y Columns (DragCoefficient, LiftCoefficient and yPlus); ¶¶ [0247]-[0257] with FIGS. 40-44: once the model is trained, its quality/accuracy can be evaluated using the test data put aside as part of preprocessing; the end-user then selects the predicted vs. actual step of the model manipulator; test data and DragCoefficient is selected as the output; this may be used to evaluate how a model is predicting unseen data. In this case, the model has high accuracy, as the points are closed to the perfect prediction line as shown in FIG. 40; another way to evaluate the model is to import data which is especially designed to validate predictions; using 'Import & Export----;,Tabular', validation files can be uploaded as a new table; in this example, the validation data is a design sweep: AngleOfAttack is the only parameter changing between each simulation; using 'Model→Validation Plot', the validation data is selected for the test data, AngleOfAttack for the X-axis and LiftCoefficient for the yAxis; FIG. 41 provides the results which shows that the predictions are accurate; an important outcome of training the model is uncertainty quantification: there's a 95% probability that measured data will be contained within the uncertainty region; this helps an end-user knowing when the predictions can be trusted; ¶ [0305]: after the neural network model has finished training, the optimal set of input parameters is found for a target set of outputs; ¶ [0689]: the resulting Monolith dashboards enabled users to upload a target dissolution profile and return an optimized particle size and shape distribution that would produce that target dissolution profile; ¶ [0703]: the performance of each gas meter design may be given by its error curve, giving the percentage accuracy for a range of flow rates and ambient temperature conditions) (KAY, ¶ [0005] and [0101]-[0102]: the outcome of the fabrication process is characterized by one or more quality metrics or criteria by which the relative success of the fabrication process can be assessed; a range or threshold of desired values of one or more such quality metrics can be provided by a user at the outset of the optimization method, and the endpoint of the method can be a set of process parameter values that produce an outcome within the desired range of the quality metrics specified at the outset).
Claims 3 and 11
RAMANATH in view of AHLFELD and KAY discloses all the elements as stated in Claims 1 and 9 respectively and further discloses wherein performing the statistical analysis to generate the plurality of uncertainty intervals based on the input parameters further comprises utilizing an automated process to select and compare surrogate models (RAMANATH, Page 33, line 12 – Page 34, line 4 with FIG. 8: upon definition of the variables, a series of prompts are generated to identify the user intent; the prompts may be generated using questionnaire templates, some examples of prompts include "Do you have a faster Analysis Driver?", "Do you want to use a metamodel or a surrogate model?", "Which physics domain are you working on (e.g. Hydraulics/ Pneumatics, Electrical, Controls, Thermodynamics, Mechanical or Chemical)?", "What is the intended study?", "Do you want to understand the variance/spread in the output response?" etc.; at step 804, the uncertainty quantification algorithm is selected by applying a decision tree to the user intent; e.g., if the user answers 'yes' to "Do you have a faster Analysis Driver?", a Direct Analysis Driver may be enabled; otherwise, if the user answers 'yes' to "Do you want to use a metamodel or a surrogate model?", then steps for setting up a surrogate model is initiated; a forward uncertainty quantification algorithm is selected from a plurality of forward uncertainty quantification algorithms such as, for example, Monte Carlo, Polynomial Chaos or Stochastic Collocation; the uncertainty quantification is executed based on the two-tank; model. upon execution, uncertainty quantification outputs are generated; the uncertainty quantification output comprises one or more variable distributions corresponding to one or more of the output variables; Page 34, line 30 – Page 36, line 5 with FIG. 11: automatic surrogate modelling of a scenario; in particular, if mathematical model corresponding to a scenario is not present in the data store 122, automatic surrogate modelling is performed in order to identify contextual data associated with the scenario; in particular, the automatic surrogate modelling is performed for enabling inverse uncertainty quantification corresponding to one or more input variables associated with the scenario; at step 1102, a set of Design of Experiments (DoE) data is received from the user; the DoE data may include one or more of a steady state configuration file, steady state measurement data, dynamic state configuration file and dynamic state measurement data obtained at step 1006 of method 1000; at step 1104, the DoE data is preprocessed to identify a plurality of key features; at step 1106, the plurality of key features identified is used for training a surrogate model; e.g., steady state surrogate modeling may be performed by training an Artificial Neural Network based on the plurality of key features obtained from the steady state configuration file and the steady state measurement data; similarly, a dynamic state surrogate modeling may be performed by training Long Short Time Memory (LSTM) model based on the plurality of key features obtained from the dynamic state configuration file and the dynamic state measurement data; in an example, one or more training parameters such as cost function, for the surrogate model may be defined by the user; in another example, if the user fails to define the one or more training parameters, the computer system 100 may use predetermined default values for the one or more training parameters; the surrogate model further generates one or more parameter estimates corresponding to one or more parameters associated with the scenario; the one or more parameter estimates comprise variable distributions corresponding to each of the one or more parameters) (AHLFELD , ¶¶ [0009]-[0011]: a lot of engineering R&D is also done using 3D simulations, such as Finite Element Analysis (FEA) or Computational Fluid Dynamics; depending on the complexity of the problem and accuracy required, 3D simulations may be costly to compute; surrogate models may then be used for evaluating computationally expensive problems; a surrogate model is a simple model that approximate complex models using training samples; however, trust issues with surrogate models also exist such as knowing when the surrogate model has made a wrong prediction, or not understanding the steps for the model obtain a result; a solution for a platform that would allow end-users to quickly build models that can predict the performance of complex engineering problems is also needed such that less testing is performed, and quality products are developed in an efficient manner; there is also a need for a 3D explainability solution that can be applied to complex 3D data, including, but not limited to, complex engineering problems; ¶ [0205] with FIG. 16: an ideal design would be in the bottom left of the plot, but the plot shows that there is a trade-off between the target variables, such as a good drag or a good lift; ¶ [0223] with FIG. 29: FIG. 29 provides a 2D point plot to take a closer look at performance trade-off of spoiler designs; as can be deduced from the plot, Lift and Drag seems correlated, in the sense that designs with good drag tends to have bad downforce, and vice versa; ¶¶ [0242]-[0257] with FIGS. 36-44: in order to get more insight into the trade-off which exists between downforce (negative LiftCoeflicient) and drag (DragCoeflicient), a new column (new parameter) is created which will be the ratio of the two using 'Transform→quick Columns' as shown in FIG. 37; MSE (Mean Square Error) is a scoring metric by which the model is iteratively refined with each new training step; ¶¶ [0258]-[0292] with FIGS. 45-58: building AI models is often an investigative process and evaluating the accuracy of models is an integral part of this process; the following examples are provided to show how to compare the quality of different AI models to identify the one which provides the best accuracy and most engineering insight for a specific use case; the platform also enables to train multiple models; we'll now compare different models to compare their accuracy; three AI models are now trained: polynomial regression, random forest regression and neural network; the different models are then compared to decide on which one is the most suitable for a specific use case; FIG. 48 provides a more quantitative assessment of model performances and shows a table listing different metrics for the error between prediction and reality, to easily compare and rank the quality of multiple models quantitatively according to scalar metrics; FIG. 49 provides the output of the step 'Model→Compare against Criteria'; this plot shows how great a portion of the test data has all predictions contained below a certain error value; the closer the curve is to the top left of the plot, the more accurate the model; AI models may also be used to solve complex optimization problems: finding the optimal set of design parameters which satisfy a target set of performance metrics; many constraints (structural dynamics, materials, maintenance cost, etc.) force aerodynamics to make trade-off decisions; ¶¶ [0342]-[0355]: Machine learning based techniques may be implemented for making models applied to 3D data explainable; explainability also may be used to better evaluate machine learning models or investigate what is happening "inside"; a surrogate model is an approximate model that can be constructed to mimic the input-output behavior of a complex simulation model as closely as possible; a surrogate model is computationally cheaper to evaluate and easy to setup as meshes are not used. Results of surrogate models are instantaneous or near instantaneous; hence surrogate models may be used to simulate the complex models in several additional scenarios, thereby multiplying the number of data points available; however, trusting a surrogate model may prove difficult as it is a black box and downstream effects are hard to see; a pure optimization model may also be less trustworthy if you just get a single design and don't understand how you got there; several features in the Monolith platform enable an end-user to gain a better understanding of how complex black-box models applied to 3D data are working) (KAY, ¶¶ [0105]-[0107] with FIG. 1: as the number of design parameters increases, the complexity of the DoE space escalates sharply, requiring advanced techniques that integrate ML with optimization methods to efficiently explore and determine the ideal synthesis parameters within the multi-dimensional design space; moreover, in the entire design space, feasible and infeasible regions denote the success or failure of the synthesis sample, respectively; clearly identifying the boundaries between these regions is crucial for minimizing the effort required in parameters exploration; introduce an adaptive experimental design strategy that synergizes a constraint learning model-implemented through a classification model-with Bayesian optimization, named Constrained BO. This integration enables efficient exploration across a multi-dimensional DoE landscape and identify optimal synthesis parameters for high-quality 2D materials; employing a dynamically updated constraint learning model, we estimate the boundary within the DoE hyperspace, distinguishing between regions where 2D monolayer crystal growth is probable those where it is likely to fail; with the "successful" regions identified, our optimization framework narrows its focus exclusively to these "successful" regions to search for the optimal experimental conditions; finally, an ML surrogate model is employed, using a statistical approximation of the target quality, coupled with a query generator based on multiple sampling criteria; this approach is used to recommend synthesis parameters for improved target quality (narrower σA) and to improve the estimation of unknown constraint function that distinguish between successful and failed conditions for monolayer crystal growth within the DoE space; through successful iterative implementation and validation of MoS2 synthesis and characterization steps, this method is able to achieve high accuracy and reliability even with limited experimental data (only 15% of the experimental data compared required for with needs of a full factorial design) and minimal additional trials suggested by the sampling recommendation algorithm; this novel approach stands out in its effectiveness in fast learning and attaining the highest possible material quality, while minimizing the need for extensive additional experiments; the overall modeling framework utilizes a closed-loop framework with two major components: (i) a constraint model 105 to identify the decision boundary between "success" and "failure" samples, and (ii) a surrogate model 110 to predict the synthesis outcome; these two components interact with each other, and their predictive outcomes guide the selection of informative samples for the subsequent batch of experiments; specifically, the constraint model 105, functioning as a binary classifier, is employed to predict the boundary that distinguishes between successful DoE parameters and unsuccessful parameters; however, the specific boundary is unknown and needs to be learned through an active learning procedure; given the prediction of successful experimental regions from the constraint model 105, the GPR-based surrogate model 110 is then built to predict the target function, capturing the relationship between the process parameters and linewidth; the surrogate model 110 (e.g., the regression model) is trained via the package GPy with the Matern5 Kernel function; combining the sampling recommendations derived from both classification and regression results, this establishes the parameters for the subsequent experiment; ¶¶ [0112]-[0116] with FIG. 1: the surrogate model may be a regression model, such as a Gaussian Process Regression model; the trained surrogate model may predict a fabrication quality metric in the form of mean value and variance; the mean value represents the most likely value of the quality metric and the variance represents the uncertainty of the predicted value; finally, a design space is defined for the particular fabrication process and the respective process variables that are being optimized; the range for each process variable may be preconfigured by a user based on user experience or theoretical physical limits or other theoretical considerations; in addition to a range of values, a step size or grid size may be defined for the design space; the step or grid size is used to determine the potential sample process variable values may be selected from the design space values; before performing the optimization process, the constraint model and surrogate model are trained using the respective constraint model training dataset and surrogate model training dataset; the feasible region is an evolving region that changes as the constraint model is trained with each iteration; ¶¶ [0123]-[0124] with FIG. 2: sampling strategy #2 aims to select data points from the design space that strike a balance between desirable target values and manageable prediction uncertainty; the surrogate model is used to predict the variance and mean for each data point in the design space that has not already been included in the processing data; for each data point in the design space, a weighted average score of the predicted mean and predicted variance is computed; the weighted average scores are ordered and the top ranked data point(s) are selected, depending on the defined sample number, as new data points (i.e., parameter values) for the next-round experiment; the purpose of sampling strategy #3 is to perform experiments for highly uncertain data points of the surrogate model and receive the true values that are used to refine the surrogate model, getting closer to the ground truth; the surrogate model is used to predict the variance value for each data point in the design space that has not already been included in the processing data; from these data points and corresponding variance values, data points are selected for refining the surrogate model; the predicted variance values may be ranked in descending order and the top ranked data point(s) are selected, depending on the defined sample number, as new data points for the next-round experiment; ¶¶ [0150]-[0191] with FIGS. 11 and 6: for the optimization framework 1100, two distinct models may be employed functioning collaboratively; the first model, a constraint model 1115, is utilized to estimate the boundaries of the feasible area within the design space; the second model, a surrogate model 1120, is designed to learn the objective function; these two models operate through two separate but interconnected feedback loops, an optimization loop 1105, and an active learning loop 1110; each loop employs its own strategy for querying new data while the active learning loop utilizes an active sampling strategy based on multiple criteria to explore the boundary for the feasible region, the optimization loop finds the global optimal within a feasible region; together, the loops collaborate not only to optimize the objective process but also to reveal the relationships between process variables (as inputs) and their resultant properties (as outputs); to find the optimal process parameters x* within feasible space, which may be regarded as a constrained optimization problem in turn, shown in equation 1; this work assumes that both f and h follow a Gaussian Process (GP) prior; there are two parts in the active learning loop 1110 in FIG. 11; a constraint model 1115 is adopted to approximate the unknown feasibility constraint, which provides an estimated feasible region varied iteratively for searching the global optimum; in addition, a novel active sampling strategy based on three sampling criteria is designed to refine the constraint learning more effectively; thereafter, a heuristic-based multi-criteria sampling strategy to identify the most informative samples in a batch mode to refine the approximation of the constraint with the least human effort is described in detail; a GP classification model may be used to learn the constraint boundary of process feasibility; an Expectation Propagation algorithm is used to approximate the posterior q(h(x)IX, C) which is non-Gaussian via the Gaussian approximation; in each iteration, the constraint model generates a predicted feasible design space Ωf for subsequent unlabeled data selection in the optimization loop; the predictive probability may also be utilized to measure the information of unlabeled data to guide data selection in constraint learning loop; in this active constraint learning loop, three criteria may be adopted jointly to quantify the information carried off each unlabeled sample: a) prediction uncertainty measures the confidence of the current classifier; b) representativeness reveals the hidden pattern of unlabeled data, and 3) diversity is introduced to maximize the information in the selected batch; there are two goals of active sampling via multiple sources of information; the first goal is to improve the optimization efficiency and the secondary goal is to learn the constraint function; considering the three criteria described above, a linear aggregated function Q(xi) is proposed to select a batch of samples from the unlabeled dataset, which is defined as Equation (10); to dynamically control the preference of each criterion in iteration t, another hyperparameter c[Symbol font/0xCE](0, l] is introduced; the weight parameter αt is decayed multiplicatively by ϵ in each iteration t and α0 is predefined as an initial value of αt; there are two components in the objective optimization loop 605 in FIG. 6; a surrogate model 620 may be adopted to capture the process-to-quality relationship with uncertainty quantification, which guides the subsequent unlabeled data selection for objective optimization within the estimated feasible design space; meanwhile, to enhance the learning capability of the surrogate model 620 with limited feasible experimental points, a pseudo-labeling technique via self-training is adopted to label unfeasible designs with predicted target quality; the next sampling point is determined by an acquisition function computed from the prediction of surrogate model training via an augmented dataset; in this optimization loop, a GP regression may be used as the surrogate process, which captures the relationship between experimental parameters and objective quality; to enhance the surrogate model's learning ability, a pseudo-labeling technique is incorporated using a self-training mechanism to enlarge the training dataset for subsequent iterative selection of unlabeled data; with the help of the surrogate model of the target process, an acquisition function is constructed to quantify the most informative candidate samples for objective optimization; the BO-ACL framework (Bayesian Optimization with Active Constraints Learning), a novel approach that synergizes two collaborative loops for querying samples; the BO-ACL framework may optimize an objective quality of interest while simultaneously learning an unknown constraint function, all with minimal human intervention; additionally, a pool-based active learning process may be used within a finite candidate pool predetermined by experimenters; the BO-ACL framework may be terminated when the global optimal sample is found and by expert knowledge; BO-ACL implements a sequential sampling approach within each batch, which ensures that each selection impacts subsequent choices within the same iteration; in each iteration, among M selections, the first sample is chosen using Algorithm 2, followed by M-1 samples selected sequentially through Algorithm 1; the optimization and active learning loop influence each other's selection processes; the BO-ACL framework is designed to optimize the target objective function with the least human effort following the feasibility constraint. The convergence of the BO-ACL framework is evaluated to the global optimum).
Claims 4 and 12
RAMANATH in view of AHLFELD and KAY discloses all the elements as stated in Claims 3 and 11 respectively and further discloses wherein the surrogate models further comprise machine learning algorithms (RAMANATH, Page 34, lines 10-33 with FIG. 11: at step 1106, the plurality of key features identified is used for training a surrogate model; e.g., steady state surrogate modeling may be performed by training an Artificial Neural Network based on the plurality of key features obtained from the steady state configuration file and the steady state measurement data; similarly, a dynamic state surrogate modeling may be performed by training Long Short Time Memory (LSTM) model based on the plurality of key features obtained from the dynamic state configuration file and the dynamic state measurement data; in an example, one or more training parameters such as cost function, for the surrogate model may be defined by the user; in another example, if the user fails to define the one or more training parameters, the computer system 100 may use predetermined default values for the one or more training parameters; the surrogate model further generates one or more parameter estimates corresponding to one or more parameters associated with the scenario; the one or more parameter estimates comprise variable distributions corresponding to each of the one or more parameters) (AHLFELD, ¶¶ [0259]-[0262]: compare the quality of different AI models to identify the one which provides the best accuracy and most engineering insight for a specific use case; three AI models are now trained: polynomial regression, random forest regression and neural network; the different models are then compared to decide on which one is the most suitable for a specific use case; ¶¶ [0662]-[0677]: shape optimization may be one of the ultimate goal for most engineering companies; the problem is finding the optimal shape for the given target function(s) and constraints; in practice, actual values of the optimized function are not known (as it would require running e.g. expensive physical simulation); thus, instead of optimizing a given function, we may approximate a target function with a ML model or surrogate mode; a first shape optimization is based on Shapenet cars autoencoder and a CNN to predict surface pressure for the generated shapes; an alternative method may also be implemented using DGCNN (Dynamic Graph CNN), where we used the same autoencoder and optimization algorithm, but instead of a CNN, a DGCNN with higher accuracy is used; the shape optimization algorithm may use a stochastic algorithm in which a 3D design is randomly modified to create a plurality of 3D designs; the plurality of 3D designs are then processed via the machine-learning model) (KAY, ¶¶ [0104]-[0107] and [0112] with FIG. 1: active learning is a subfield of ML-an iterative learning method which can start with a small initial dataset; under the active learning framework, Bayesian Optimization (BO) is a kind of global optimization method that leverages probabilistic ML, such as Bayesian regression and Gaussian Process, in order to find the global optimum with the least experimental cost; an ML surrogate model is employed, using a statistical approximation of the target quality, coupled with a query generator based on multiple sampling criteria; the GPR-based surrogate model 110 is built to predict the target function, capturing the relationship between the process parameters and linewidth; the surrogate model 110 (e.g., the regression model) is trained via the package GPy with the Matern5 Kernel function; the surrogate model may be a regression model, such as a Gaussian Process Regression model; ¶¶ [0139]-[0149] with FIG. 9: the proposed framework, as illustrated in FIG. 9, consists of two main components: (q) objective optimization and (b) active constraint learning (ACL) loop; the optimization loop learns the process-to-quality relationship and proposes queries toward global optimization; to improve the proficiency of this framework in simultaneously optimizing the target process and learning constraints, an active sampling strategy is devised that is founded on three key measurements: a) representativeness, b) uncertainty, and c) diversity; this strategy employs a unified function in the active loop to select the most informative unlabeled samples; a novel parameter design method is introduced that leverages both active learning and Bayesian Optimization to effectively explore the undefined constraint boundary while optimizing the process concurrently; ¶ [0150] with FIG. 11: for the optimization framework 1100, two distinct models may be employed functioning collaboratively; the first model, a constraint model 1115, is utilized to estimate the boundaries of the feasible area within the design space; the second model, a surrogate model 1120, is designed to learn the objective function; these two models operate through two separate but interconnected feedback loops, an optimization loop 1105, and an active learning loop 1110; each loop employs its own strategy for querying new data while the active learning loop utilizes an active sampling strategy based on multiple criteria to explore the boundary for the feasible region, the optimization loop finds the global optimal within a feasible region; together, the loops collaborate not only to optimize the objective process but also to reveal the relationships between process variables (as inputs) and their resultant properties (as outputs); ¶ [0176]: to enhance the surrogate model's learning ability, a pseudo-labeling technique is incorporated using a self-training mechanism to enlarge the training dataset for subsequent iterative selection of unlabeled data).
Claims 5 and 13
RAMANATH in view of AHLFELD and KAY discloses all the elements as stated in Claims 3 and 11 respectively and further discloses wherein utilizing the automated process to select and compare the surrogate models further comprises: utilizing a dual objective pareto optimal surrogate model selection process (AHLFELD, ¶ [0205] with FIG. 16: an ideal design would be in the bottom left of the plot in FIG. 16, but the plot shows that there is a trade-off between the target variables, such as a good drag or a good lift; ¶ [0223] with FIG.29: FIG. 29 provides a 2D point plot to take a closer look at performance trade-off of spoiler designs; as can be deduced from the plot, Lift and Drag seems correlated, in the sense that designs with good drag tends to have bad downforce, and vice versa; an ideal spoiler would then be located somewhere at the bottom left of the plot; ¶¶ [0242]-[0257] with FIGS. 36-44: the steps for building an AI model using as an example CFD simulations on spoiler designs for a race car; in order to get more insight into the trade-off which exists between downforce (negative LiftCoeflicient) and drag (DragCoeflicient), a new column (new parameter) is created which will be the ratio of the two using 'Transform→quick Columns' as shown in FIG. 37; Ratio is given as the new column name, and the operation is set to be the ratio of LiftCoeflicient (first column) and DragCoeflicient (second column); as well as giving predictions of performance for a new selection of designs, the platform is also capable of providing optimal designs for a selection of target variables or performance metrics; the end-user then selects 'Apply→Min/Max Optimization' as shown in FIG. 44; this algorithm will use the Neural Network model to search for the best Lift-to-Drag ratio and will return the set of design parameters corresponding to the most efficient spoiler wing; this tool may be used for quickly identifying the best possible combination of design parameters which satisfy a target variable or performance; ¶¶ [0258]-[0282] with FIGS. 47-55: building AI models is often an investigative process and evaluating the accuracy of models is an integral part of this process; how to compare the quality of different AI models to identify the one which provides the best accuracy and most engineering insight for a specific use case; the different models are then compared to decide on which one is the most suitable for a specific use case; FIG. 47 provides the output of the validation plot that shows the 'ground truth' data (outcome of a simulation) overlaid with predictions from each model; FIG. 48 provides a more quantitative assessment of model performances and shows a table listing different metrics for the error between prediction and reality, to easily compare and rank the quality of multiple models quantitatively according to scalar metrics; by rotating the response surface around in FIG. 51, it is possible to quickly identify how this trade-off exists; AI models may also be used to solve complex optimization problems, finding the optimal set of design parameters which satisfy a target set of performance metrics; as an example, numerical simulations of the performance of wind turbine designs may be used; they are complex engineering products which present interesting design optimization challenges; their main function is to convert wind energy into electricity as efficiently as possible; however, many constraints (structural dynamics, materials, maintenance cost, etc.) force aerodynamics to make trade-off decisions; two different models are now trained: one to predict the scalar performance metrics, and the other to predict the axial velocity; ¶¶ [0662]-[0677]: multi-criteria optimization: a multiple target functions may also be optimized, such as by using pareto front optimization approach) (KAY, ¶ [0123]: sampling strategy #2 aims to select data points from the design space that strike a balance between desirable target values and manageable prediction uncertainty; ¶ [0129]: the ML guided experimental iterations were performed in two stages; iterations in stage 1 were performed to find the optimized synthesis condition (DoE parameters) for the narrowest linewidth in the fastest possible time; this is the "brachistochrone" stage; once linewidth obtained from recommended design reach to the theoretical minimum value (around 38 meV), it moves into stage 2; this second stage conducts cross-validation of the optimization by proposing DoE values that deviate from the optimized condition; this is done to test the robustness of our classification accuracy and prediction error, examining if better designs exist in both "near" and "away" from the most optimized DoE settings; these recommendations further affirm the robustness of the constraint model's accuracy by evaluating the prediction error of linewidth as determined by the regression model; ¶¶ [0134] and [0176]-[0178]: the acquisition function controls the selection of unexperimented DoE parameters via balancing low linewidth prediction (exploitation) and high prediction uncertainty (exploration); this evolution aims to ensure that our sampling strategy sufficiently explores all regions where the optimal DoE could potentially exist, leveraging predictions from our regression mode; the Upper Confidence Bound (UCB) acquisition function may be used to minimize the target quality of production, the acquisition function is designed as Equation (19), where γ is a hyperparameter that controls the trade-off between these two terms; ¶ [0137]: introduce an adaptive and sequential experimental design framework, utilizing machine learning to rapidly guide the synthesis of MoS2 towards the highest possible quality with only 15% of the experimental data required by the traditional full factorial design method, and experimentally validates the effectiveness of this framework; overcome the uncertainties involved in a multi-parameter 2D synthesis design space and quickly arrive at a specific outcome; validate the global minimum (within the tested range of DoE parameters) through a balance of "exploration" vs. "exploitation" steps); processing assumptions using hyper-parameter optimization before performing the statistical analysis on the input parameters (AHLFELD , ¶¶ [0209]-[0215]: several Neural Network parameters may also be either user-configurable or automatically determined by the platform: (a) Number of Hidden Layers & Hidden Layer Size: one important choice is the shape of the network, wherein (i) the neurons in a Neural Network are arranged in a series of layers known as "hidden layers"; (ii) the number and the size of these hidden layers may be easily configurable by an end-user; (iii) both choices affect the model performance: a model with more layers (or larger layers) will be able to fit complex relationships but is more liable to overfitting; (iv) a model with fewer layers (or smaller layers) will only be able to capture simpler relationships but is less prone to overfitting; (v) in general, finding the best choice is a balance between these two extremes (underfitting and overfitting); and (vi) multiple machine learning models can be trained and compared with different architectures, and the performance both on training data and on unseen testing data is determined and displayed to an end-user; (b) number of training steps: another important choice is how many training steps to use, wherein (i) using more training steps will make the model fit more closely to the training data; (ii) it is usually a good thing to make the model predictions match the training data more closely; (iii) however, if a model is trained for too many steps then the training data can be overfitted; (iv) overfitting means that the model makes good predictions for training data, but worse predictions for unseen data; (v) the platform may be configured to automatically determine an optimum number of training steps; and (vi) further loss history curves may also be determined and displayed to enable an end-user to decide how many training steps to use; (c) Batch Size: Neural Networks look at the data in chunks or 'batches', wherein the number of rows in each batch ('Batch Size') can affect how quickly the model trains and can also affect the final performance; (d) Dropout Fraction: this parameter can be used to control the randomness in the network, wherein (i) a higher dropout fraction increases the randomness in the model, by randomly turning off ('dropping out') some fraction of the neurons in each training step; and (ii) higher dropout fraction can help reduce the chance of overfitting (through a process known as 'regularization'), but it can mean that the model will require more training steps to learn to fit the data; and (e) Uncertainty: Neural Networks can provide uncertainty estimates along with their predictions, which may only be possible if the Dropout Fraction has a non-zero value; when all the parameters are entered or selected, a plot is then provided with the result of loss function as a function of the number of training steps, as shown in FIG. 19; ¶¶ [0250]-[0252] with FIG. 42: the platform provides an understanding of how data volumes affect the accuracy of machine learning models; the Neural Network is twice as accurate when trained on 80% of the data as compared to 20% of it, as shown in FIG. 42; advantageously, this provides a helpful tool for an end-user to understand how much more data is needed to reach a specific accuracy); and utilizing a statistical test to analyze the performance of the surrogate models (RAMANATH, Page 35, line 19 – Page 36, line 32 with FIGS. 11-12 and 13A-C: at step 1106, the plurality of key features identified is used for training a surrogate model; e.g., steady state surrogate modeling may be performed by training an Artificial Neural Network based on the plurality of key features obtained from the steady state configuration file and the steady state measurement data; similarly, a dynamic state surrogate modeling may be performed by training Long Short Time Memory (LSTM) model based on the plurality of key features obtained from the dynamic state configuration file and the dynamic state measurement data; in an example, one or more training parameters such as cost function, for the surrogate model may be defined by the user; in another example, if the user fails to define the one or more training parameters, use predetermined default values for the one or more training parameters; the surrogate model further generates one or more parameter estimates corresponding to one or more parameters associated with the scenario; at step 1202, variable distributions estimated through the uncertainty quantification is subjected to a non-parametric test such as Kolmogorov-Smirnov test) (AHLFELD, ¶¶ [0206]-[00208] with FIGS. 17-18: training a model means building a mathematical model which will be able to correlate the simulation inputs to its outputs; the end-user then selects all the input parameters in the 'Inputs' dropdown menu, then selects both output parameters in the 'Outputs' fields, ticks the 'Include Uncertainty' checkbox, and clicks 'Apply' as shown in FIG. 18; machine learning models include many parameters that the end-user can use to tune the prediction performance of a model; sensible default choices for all these parameters are automatically selected, so the end-user can just press Apply and train a model immediately; ¶¶ [0244]-[0249] with FIGS. 39-41: an AI model may also be trained on a subset of the training data (such as about 80%) and the remaining 20% of the training data is used as validation data to evaluate the model's quality; uncertainty is included to ensure that each prediction is accompanied by a measure of uncertainty; after the end-user clicks apply, a graph is displayed; the graph displayed is continuously updating as the Neural Network is being trained as shown in FIG. 39; MSE (Mean Square Error) is a scoring metric by which the model is iteratively refined with each new training step; once the model is trained, its quality/accuracy can be evaluated using the test data put aside as part of preprocessing; the end-user then selects the predicted vs. actual step of the model manipulator; test data and DragCoefficient is selected as the output this may be used to evaluate how a model is predicting unseen data; using 'Model→Validation Plot', the validation data is selected for the test data, AngleOfAttack for the X-axis and LiftCoefficient for the yAxis; FIG. 41 provides the results which shows that the predictions are accurate; an important outcome of training the model is uncertainty quantification: there's a 95% probability that measured data will be contained within the uncertainty region; this helps an end-user knowing when the predictions can be trusted; ¶¶ [0253]-[0255] with FIG. 43: the end-user then selects 'Curve Prediction' as shown in FIG. 43. AngleOfAttack is selected as the input and DragCoefficient as the output; the results suggest that the uncertainty will be higher for higher values of AngleOfAttack; ¶¶ [0275]-[0283] with FIG. 55: the design selection can be varied in the Fix Parameters step and the instant prediction of performance is provided in the two subsequent steps, as shown in FIG. 55; a trained model can also be used with a dataset prediction step to perform virtual testing; looking at engine calibration data as an example, the platform may be used to assess if reducing the charging loss ( due to friction within the gas chamber) by 5% would impact the fuel consumption).
Claims 7 and 15
RAMANATH in view of AHLFELD and KAY discloses all the elements as stated in Claims 1 and 9 respectively and further discloses wherein generating the report including the plurality of uncertainty intervals and future test point recommendations comprises generating the report based on active learning (AHLFELD , FIG. 20: options for "Optimization" ("Find the best performing designs") shown in FIG. 20 including (a) Active Learning; (b) Min/Max Optimization; (c) Targeted Optimization; ¶ [0066] and [0256] with FIG. 44: FIG. 44 shows a screenshot with an example of output of the 'Min/Max Optimization' feature; ¶¶ [0090]-[0091] and [0304]-[0311] with FIGS. 68-69: FIG. 68 shows the output displaying the results of the optimization step displaying three optimal set of input parameters found for a target set of outputs; FIG. 69 shows a screenshot in which the end-user can scroll up to the Targeted Optimization step results and generate a new design; ¶ [0020]: for new car design, the Monolith platform uses self-learning models to instantly predict the results of complex vehicle dynamics systems, in the process reducing the need for physical tests or simulations; the platform leverages this data to train AI self-learning models to accurately predict performance; by interpreting vehicle systems' behavior from the data, the AI helps engineers understand how vehicles react in areas that are currently impossible to simulate, as well as under different conditions; ¶¶ [0147]-[0159]: engineering data is used to create accurate, self-learning models to quickly understand and instantly predict the performance of complex systems under many operating conditions; self-learning models are built with test data from all measurement, signal or sensor data types including time-series, 3D CAD and tabular data; self-learning models can be improved by feeding more data, hence more learning and less testing is needed to develop quality product faster; ¶¶ [0668]-[0691]: Pharmaceutical Development Using Self-Leaming Models; ¶¶ [0692]-[0706]: Smart Meters Using Self-Leaming Models: (a) create accurate, self-learning models to quickly understand and instantly predict the performance of the gas meter system under thousands of operating conditions; (b) determine what smart meter designs give the best performance, while self-learning models learn from new data generated along the way, indicating which designs are most promising to investigate next; and (c) predict error curve response for any new set of design choices, specified by dragging sliders which empowers any engineer to interact with the model and gain new insights, with no need for programming expertise, IT set-up, and minimal training to use the platform and build self-learning models) (KAY, ¶¶ [0104], [0107], and [0112] with FIG. 1: active learning is a subfield of ML-an iterative learning method which can start with a small initial dataset; under the active learning framework, Bayesian Optimization (BO) is a kind of global optimization method that leverages probabilistic ML, such as Bayesian regression and Gaussian Process, in order to find the global optimum with the least experimental cost; the constraint model 105, functioning as a binary classifier, is employed to predict the boundary that distinguishes between successful DoE parameters and unsuccessful parameters; however, the specific boundary is unknown and needs to be learned through an active learning procedure; ¶¶ [0139]-[0150], [0154], and [0158] with FIGS. 9 and 11: the proposed framework, as illustrated in FIG. 9, consists of two main components: (q) objective optimization and (b) active constraint learning (ACL) loop; the optimization loop learns the process-to-quality relationship and proposes queries toward global optimization; the ACL loop learns the constraint boundary of process feasibility and chooses the most informative samples to be labeled in the next round; the ACL loop estimates the feasible region iteratively, which adapts over time to assist in optimizing the target process; similarly, the query proposed from the optimization loop will also influence the selection in the ACL loop; a constraint model may be introduced within the ACL loop to learn about feasibility constraints; Gaussian Process (GP), a widely used constraint model, may evaluate prediction uncertainty; a standard AL strategy queries the unlabeled sample near the classification boundary with the highest prediction uncertainty; to improve the proficiency of this framework in simultaneously optimizing the target process and learning constraints, an active sampling strategy is devised that is founded on three key measurements: a) representativeness, b) uncertainty, and c) diversity; this strategy employs a unified function in the active loop to select the most informative unlabeled samples; a novel parameter design method is introduced that leverages both active learning and Bayesian Optimization to effectively explore the undefined constraint boundary while optimizing the process concurrently; active learning is widely adopted in machine learning applications when the labeling process is costly; uncertainty and diversity are two of the most adopted criteria of AL for selecting unlabeled samples; uncertainty-based methods, such as query-by-committee and entropy-based sampling, choose unlabeled data with the most uncertain prediction to annotate labels for model retraining; meanwhile, diversity represents the similarity between labeled and unlabeled data; diversity-based methods choose the most dissimilar samples to query via different measurements, such as Euclidean distance and Cosine similarity; considering multiple criteria simultaneously can improve AL performance; enhance the AL by implementing a dynamic weighted multi-criteria sampling strategy; as a key element of the framework, the active sampling method for the constraint model plays a significant role in improving decision boundary exploration; it also collaborates with the optimization process, thereby increasing the rate of optimization speed; for the optimization framework 1100, two distinct models may be employed functioning collaboratively; the first model, a constraint model 1115, is utilized to estimate the boundaries of the feasible area within the design space; the second model, a surrogate model 1120, is designed to learn the objective function; these two models operate through two separate but interconnected feedback loops, an optimization loop 1105, and an active learning loop 1110; each loop employs its own strategy for querying new data while the active learning loop utilizes an active sampling strategy based on multiple criteria to explore the boundary for the feasible region, the optimization loop finds the global optimal within a feasible region; together, the loops collaborate not only to optimize the objective process but also to reveal the relationships between process variables (as inputs) and their resultant properties (as outputs); there are two parts in the active learning loop 1110 in FIG. 11; a constraint model 1115 is adopted to approximate the unknown feasibility constraint, which provides an estimated feasible region varied iteratively for searching the global optimum; in addition, a novel active sampling strategy based on three sampling criteria is designed to refine the constraint learning more effectively; in this active constraint learning loop, three criteria may be adopted jointly to quantify the information carried off each unlabeled sample; in these three criteria, a) prediction uncertainty measures the confidence of the current classifier; b) representativeness reveals the hidden pattern of unlabeled data, and 3) diversity is introduced to maximize the information in the selected batch; there are two goals of active sampling via multiple sources of information; the first goal is to improve the optimization efficiency and the secondary goal is to learn the constraint function; ¶¶ [0179]-[0180]: introduce the BO-ACL framework (Bayesian Optimization with Active Constraints Learning), a novel approach that synergizes two collaborative loops for querying samples; the BO-ACL framework may optimize an objective quality of interest while simultaneously learning an unknown constraint function, all with minimal human intervention; additionally, a pool-based active learning process may be used within a finite candidate pool predetermined by experimenters; the initial dataset in Algorithm 3 is selected via the Latin Hypercube Sample (LHS) method; the BO-ACL framework may be terminated when the global optimal sample is found and by expert knowledge; unlike traditional batch model sampling strategies, BO-ACL implements a sequential sampling approach within each batch; this method ensures that each selection impacts subsequent choices within the same iteration; in each iteration, among M selections, the first sample is chosen using Algorithm 2, followed by M-1 samples selected sequentially through Algorithm 1; the optimization and active learning loop influence each other's selection processes; for a detailed understanding, the pseudo-code of the BO-ACL framework is outlined in Algorithm 3).
Claims 8 and 16
RAMANATH in view of AHLFELD and KAY discloses all the elements as stated in Claims 1 and 9 respectively and further discloses wherein generating the plurality of uncertainty intervals further comprises: generating a coverage plot bounded over one of the plurality of uncertainty intervals; and displaying a regression response with associated uncertainty interval on a graphical user interface (RAMANATH, Page 33, lines 12-21: upon definition of the variables, a series of prompts are generated to identify the user intent; the prompts may be generated using questionnaire templates, some examples of prompts include "Do you have a faster Analysis Driver?", "Do you want to use a metamodel or a surrogate model?", "Which physics domain are you working on (e.g. Hydraulics/ Pneumatics, Electrical, Controls, Thermodynamics, Mechanical or Chemical)?", "What is the intended study?", "Do you want to understand the variance/spread in the output response?" etc.; Page 36, line 7 – Page 38, line 15 with FIGS. 12 and 13A-C: generating a user-interpretable report based on an outcome of performing an uncertainty quantification; the user-interpretable report is a summarized form of the variable distributions associated with the uncertain variables determined through the uncertainty quantification; at step 1202, variable distributions estimated through the uncertainty quantification is subjected to a non-parametric test such as Kolmogorov-Smirnov test; e.g., if three variable distributions corresponding to the variable Kvs-valvel is obtained as shown in FIG 13A-C, the non-parametric test may identify the actual variable distribution for the variable Kvs-valvel from the three distributions based on goodness of fit value i.e., the distribution with minimum chi-square value; the different types of distributions along with the chi-square values are provided to the right-hand side of each estimated variable distribution as shown in the FIGS 13A-C; in particular, FIG 13A shows a Graphical User Interface (GUI) displaying a triangular distribution of MLV = 0.2935866608, lower limit = -3.0360675907, upper limit = 11.0184763; FIG 13B shows a GUI displaying a normal distribution of mean = 0.2590632, standard deviation = 0.05080291; FIG 13C shows a GUI displaying a uniform distribution of lower limit = 0.0521730, upper limit = 0.259770349; at step 1204, one or more key parameters associated with the identified variable distribution is derived using a predefined library, for example, SciPy; at step 1206, the one or more key parameters are embedded into a natural language text using a natural language generation (NLG) technique) (AHLFELD , ¶¶ [0262]-[0265] with FIGS. 47-49: compare different models to compare their accuracy; three AI models are now trained: polynomial regression, random forest regression and neural network; the different models are then compared to decide on which one is the most suitable for a specific use case; FIG. 47 provides the output of the validation plot that shows the 'ground truth' data (outcome of a simulation) overlaid with predictions from each model; as would be expected for the oscillating response, the results from the Polynomial Regression model are poor; the Random Forest Regression model and the neural network, however, capture the non-linear physical trends; FIG. 48 provides a more quantitative assessment of model performances and shows a table listing different metrics for the error between prediction and reality, to easily compare and rank the quality of multiple models quantitatively according to scalar metrics; FIG. 49 provides the output of the step 'Model→Compare against Criteria'; this plot shows how great a portion of the test data has all predictions contained below a certain error value; the closer the curve is to the top left of the plot, the more accurate the model; in this example, only the neural network model satisfies the criteria; ¶ [0206] with FIG. 7: the end-user clicks on the 'Model' manipulator and the 'Model' features are then displayed as shown in FIG. 17; in this example, the end-user selects 'Neural Network'; other selection options are Decision Tree Regression, Gaussian Process Regression, Linear Regression; Nearest Neighbors Regression; Polynomial Regression; Random Forest Regression, Support Vector Regression; ¶ [0205] with FIGS. 15-16: the end-user selects '2D Point plot'; the end-user then selects the axes and the display options using one or more variables from the original dataset, as shown in FIG. 15; . the end-user chooses 'Output:DragCoeflicient' for the X column, 'Output:LiftCoeflicient' for the Y column, and 'Input:AngleOfAttack' for the Marker color and clicks 'Apply'; FIG. 16 shows the plot obtained; an ideal design would be in the bottom left of the plot, but the plot shows that there is a trade-off between the target variables, such as a good drag or a good lift; ¶¶ [0222]-[0223] with FIGS. 28-29: the distribution of 'DragCoeffectient' and 'LiftCoefficient' is displayed in FIG. 28; FIG. 29 provides a 2D point plot to take a closer look at performance trade-off of spoiler designs; ¶¶ [0232]-[0241] with FIGS. 31-35: FIG. 31 provides the output of the intelligent correlations step 311: a heat map with the color intensity representing the strength of correlation for the strongest relationship found between any pair of variables; the end-user can also interact with the heatmap by hovering and/or clicking on a single square, and a scatterplot with the associated line of best fit will appear below the heatmap for the two variables that are represented when the end-user clicks on a square; enable an end-user to quickly understand the relationships between multiple different variables, such as 5 or 10 different variables, using a parallel coordinates; ¶¶ [0260]-[0261] with FIGS. 45-46: FIG. 45 shows the results of the step 'Explore→Line plot'; the pitch angle is plotted as a function of time showing all the training data: 100 different dynamic responses to a gust, for different aircraft characteristics; FIG. 46 provides the output of the 'Explore→Parallel Coordinates' for the newly created table; the data points plotted are the single lowest values of pitch angle over time for each simulation; ¶¶ [0266]-[0267] with FIGS. 50-51: FIG. 50 provides the output of the step 'Apply→Curve Prediction' with the neural network model selected to plot pitch angle against time; FIG. 51 provides the output of the step 'Apply→Surface Prediction'; the resultant plot is like the curve prediction tool, except with an extra input dimension; by rotating the response surface around, it is possible to quickly identify how this trade-off exists; ¶¶ [0274]-[0284] with FIG. 54: FIG. 54 shows the output of the line plot tool in which the axial velocity is displayed as a function of distance; the plots obtained show the deceleration of the flow as the air approaches and passes the turbine (located at Distance=0); the design selection can be varied in the Fix Parameters step and the instant prediction of performance is provided in the two subsequent steps, as shown in FIG. 55; the results are provided in FIG. 56 in which the output of the 'Explore-3D Point Plot' step is provided; the 3D plot displays fuel ratio as a function of engine torque and engine speed) (Kay, ¶¶ [0072]-[0074], [0114], [0121], and [0123], with FIGS. 3A-C: FIG. 3A is an illustration of an example three-parameter design space with a defined grid size for each parameter that subsequently identifies the sampling set of the design space (as illustrated by the dots); as shown in FIG. 3C, two parameters were selected for the k+1 iteration that fell near boundary, thus the actual experimental result for these parameters was uncertain; FIG. 3B illustrates the variance of predictions (shown as the shaded areas) of the surrogate model output (y-axis) given the input values (x-axis)).
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over RAMANATH in view of AHLFELD and KAY as applied to Claims 5 and 13 respectively above, and further in view of Easum et al. ("Efficient Multiobjective Antenna Optimization With Tolerance Analysis Through the Use of Surrogate Models", IEEE TRANSACTIONS ON ANTENNAS AND PROPAGATION, VOL. 66, NO. 12, DECEMBER 2018, pp. 6706-6715), hereinafter Easum.
Claims 6 and 14
RAMANATH in view of AHLFELD, and KAY discloses all the elements as stated in Claims 5 and 13 respectively and further discloses wherein the dual objective pareto optimal surrogate model selection process is based on root mean square error (AHLFELD , ¶ [0246]: MSE (Mean Square Error) is a scoring metric by which the model is iteratively refined with each new training step; ¶¶ [0264]-[0265] with FIGS. 48-49: FIG. 48 provides a more quantitative assessment of model performances and shows a table listing different metrics for the error between prediction and reality, to easily compare and rank the quality of multiple models quantitatively according to scalar metrics; FIG. 49 provides the output of the step 'Model→Compare against Criteria'; this plot shows how great a portion of the test data has all predictions contained below a certain error value; the closer the curve is to the top left of the plot, the more accurate the model; ¶¶ [0470]-[0474] with FIG. 99: FIG. 99 provides the results in which actual (left) and predicted (right) surface pressure on the test set are compared using mean squared error (MSE); we average the predictions per point; this can be also used to estimate uncertainty or risk by computing standard deviation for each point. Alternatively, stochastic dropout or bootstrap may also be used; the method may also be used to estimate uncertainties by making predictions for different subsets of input points and estimating the standard deviation of prediction for each point; ¶ [0500]: for training we minimize MSE in PCA space (sensitive to the number of pca components); ¶¶ [0636]-[0644]: during the main training we minimize reconstruction error between voxel input and its reconstruction; the reconstruction error is MSE between implicit fields, not point coordinates; there are more ways to compare original and the reconstructed error. For different applications different measures may be more suitable: Mean square error (MSE), chamfer distance, Intersection over Union (IoU), Wasserstein distance or point clouds, and Light field descriptor (LFD)) (KAY, ¶ [0123]: sampling strategy #2 aims to select data points from the design space that strike a balance between desirable target values and manageable prediction uncertainty; ¶ [0129]: the ML guided experimental iterations were performed in two stages; iterations in stage 1 were performed to find the optimized synthesis condition (DoE parameters) for the narrowest linewidth in the fastest possible time; this is the "brachistochrone" stage; once linewidth obtained from recommended design reach to the theoretical minimum value (around 38 meV), it moves into stage 2; this second stage conducts cross-validation of the optimization by proposing DoE values that deviate from the optimized condition; this is done to test the robustness of our classification accuracy and prediction error, examining if better designs exist in both "near" and "away" from the most optimized DoE settings; these recommendations further affirm the robustness of the constraint model's accuracy by evaluating the prediction error of linewidth as determined by the regression model; ¶¶ [0134] and [0176]-[0178]: the acquisition function controls the selection of unexperimented DoE parameters via balancing low linewidth prediction (exploitation) and high prediction uncertainty (exploration); this evolution aims to ensure that our sampling strategy sufficiently explores all regions where the optimal DoE could potentially exist, leveraging predictions from our regression mode; the Upper Confidence Bound (UCB) acquisition function may be used to minimize the target quality of production the acquisition function is designed as Equation (19), where γ is a hyperparameter that controls the trade-off between these two terms; ¶ [0137]: introduce an adaptive and sequential experimental design framework, utilizing machine learning to rapidly guide the synthesis of MoS2 towards the highest possible quality with only 15% of the experimental data required by the traditional full factorial design method, and experimentally validates the effectiveness of this framework; overcome the uncertainties involved in a multi-parameter 2D synthesis design space and quickly arrive at a specific outcome; validate the global minimum (within the tested range of DoE parameters) through a balance of "exploration" vs. "exploitation" steps).
RAMANATH in view of AHLFELD and KAY fails to explicitly disclose wherein the dual objective pareto optimal surrogate model selection process is based on root mean square error and a calibration area.
Easum teaches a system and a method relating to optimization with tolerance/uncertainty analysis (Easum, TITLE), wherein the dual objective pareto optimal surrogate model selection process is based on root mean square error and a calibration area. (Easum, Abstract of Page 6706: an efficient, black-box multi-objective optimization technique is presented, which is capable of simultaneously optimizing designs for performance as well as robustness when input tolerance values are not known a priori; during the optimization process, adaptive statistical surrogate mappings between input variables and output objectives are formulated within a model selection framework; these statistical models can be evaluated in fractions of a second and serve as an efficient surrogate for a more computationally intensive process, such as an electromagnetic simulation; by exploiting the speed offered from surrogate modeling techniques, new, high-performance designs can be quickly identified; in addition, complete tolerance analysis can be conducted within the optimization loop, which provides designers with critical information regarding the robustness of designs; Section I of Pages 6706-6707: the ability to assess the robustness of a design, or its sensitivity to changes in its parameters, is a critical step in the engineering design process; identifying designs that simultaneously have acceptable performance and are robust to small changes in its input parameters is a practical and inherently multi-objective design problem, i.e., there are often tradeoffs between performance and robustness; surrogate modeling is a statistical approach to global optimization which relies on empirically training a mathematical model that mimics the underlying physics of a problem under consideration; a benefit of surrogate modeling is that the local design space around a nominal solution can be quickly evaluated without the need for a computationally expensive simulation; however, in some design scenarios, acceptable input tolerance values are not known a priori, as is the case when manufacturing processes have not been predetermined; therefore, a new design technique is needed to address this important class of problems; present an efficient optimization scheme based on surrogate modeling, which is capable of performing multi-objective optimization while simultaneously assessing the robustness and determining acceptable input tolerances of designs; Section II with FIG. 1 of Page 6707: in order to formulate an accurate statistical mapping between the input and output parameters of arbitrary problems, a series of statistical regressions is performed and an automated model selection scheme is implemented to choose the best fitting model; several automated surrogate selection and validation methods of varying complexities have been proposed, while the method implemented here is relatively standard with splitting of the data into training and testing sets and evaluation metrics of mean square error (mse) and the Akaike information criterion (AIC); as shown in Fig. 1, the model selection process begins by randomly splitting the input and output data into training and testing sets; the polynomial model selection occurs by first iteratively performing a series of multivariate polynomial regressions of orders 1 through m on the training data set, where each polynomial regression includes all combinations of interaction terms in the regression; the polynomial model with the lowest AIC score is chosen as the best fitting polynomial model R1(x); next, the Kriging and GPR methods are applied to the training data set to yield the Kriging model R2(x) and the GPR model R3(x); these three models are then compared by predicting the outputs from the testing data set; the mse is computed for each method, and the method with the lowest mse is selected as the best fitting surrogate model for the data set as described in the equation (1); in order to utilize all of the data available, the best fitting model is then retrained with all N designs to yield the surrogate model that can predict one output from a set of arbitrary inputs; once a particular surrogate model has been trained and selected, evaluating the model at arbitrary points is exceptionally fast—a typical surrogate function evaluation may take on the order of hundredths or thousandths of a second to compute; Section III with FIG. 2 of Page 6707-6709: the tolerance of a design can be described in various ways but can be separated into several categories: a) known input tolerance values; b) known output tolerance values; c) mixture of known input and output tolerances; the method presented here is designed to solve problems of case 2, where no input tolerances are known a priori; here, designs of varying performance and robustness are displayed on a multi-objective Pareto front, which enables the designer to manually choose a candidate that best fits the design scenario; we call the metric utilized to assess the robustness the tolerance hypervolume, and it is directly related to the tolerance values placed on the design parameters; the tolerance hypervolume Thv can be defined as Equation (2); by numerically searching for the values of Δxi that maximize Thv, such that all of the design space within the bounding box is within the acceptable range of performance, then the input tolerances can be explicitly specified and Thv can be defined; designs with relatively large tolerance hypervolumes correspond to robust designs (i.e., they are tolerant to deviations in the input parameters), whereas designs with small tolerance hypervolumes correspond to sensitive designs; a key assumption with the tolerance hypervolume metric is that all input design parameters are assumed to be uncorrelated; otherwise, tolerance estimates cannot be expressed in a simple ±Δx format; values of Δx are numerically searched to satisfy the Equation (3); by generalizing the allowable output tolerance Δf to take the form of a vector of length T values, the user can set different acceptable tolerance values on each output objective; this technique is useful because arbitrary designs can be compared on an equivalent metric of robustness; to numerically implement this, the trained surrogate model is sampled using a space filling technique, such as the Latin Hypercube Sampling (LHS) at ns points, where the points are centered around the nominal point X; Next, each sample point is assumed to be a corner point of the bounding tolerance box, and the box is assessed if there are any interior sample points that fall outside of the
F
^
±
∆
f
bounds; the hypervolume of boxes that contain only valid sample points are then calculated with (4), and the box with the largest hypervolume is chosen as the ideal tolerance hypervolume box; to increase the accuracy of hypervolume estimate, this process is reiterated with the search area reduced to 120% of the previously estimated box size; this step essentially zooms in on the solution and can be performed q times or until a convergence criterion is met; an overview of this process is depicted in Fig. 2 with an example two-variable, single-objective surrogate-estimated response surface and q = 2 iterations; this technique is useful because arbitrary designs can be compared on an equivalent metric of robustness; in addition, this process enables input tolerance values to be estimated for any point in the design space and communicates the results in a commonly utilized format of symmetric tolerance values; furthermore, the tolerance values take into account simultaneous deviations in multiple input parameters, which can be difficult to obtain with a technique, such as parameter sweeping; it should be noted that the tolerance values listed are only estimates, and the accuracy of these estimates is dependent on the quality of the surrogate mapping, the number of samples used to create that map, and the locations of the samples; After completing the multi-objective optimization, it is recommended that the tolerance estimate is validated with the full simulation by using a method, such as a parameter sweep, or manual spot checks in order to assess the accuracy of the tolerance estimates; Section IV of Page 6709-6710: by integrating the surrogate modeling and tolerance analysis techniques described into an optimization framework, high-performance and robust designs can be identified; after the user defines input constraints, output objectives, allowable output tolerances, initial population size, and the number of samples between retraining, then the optimization occurs with the following steps: a) sampling of the design space with the high-fidelity cost function; b) training and selection of surrogate models (as described in Section II); c) estimation of the tolerance hypervolume for all previously sampled designs; d) searching surrogate cost function for single-objective optima (without tolerance estimation); e) seeding a multi-objective optimization with surrogate-estimated optimal designs; 6) searching surrogate cost function with multi-objective optimization and tolerance estimation; 7) validation of several designs estimated to be Pareto optimal with the high-fidelity model; 8) check if the stopping criteria has been met—if not return to step 1; since polynomial regressions are implemented in the surrogate model selection framework suggested previously, at least p + 1 initial samples are needed; additional samples are added in an adaptive sampling strategy, which balances between exploration and exploitation of the design space; exploration refers to adding sample points in untested regions of the design space with a strategy, such as LHS; exploitation refers to adding sample points in regions of the design space, which are estimated to be optimal; a total of nnew samples are added in each round of retraining after the initial population, such that nnew = nexploit + nexplore (4), where nexploit is the number of new exploitation samples and nexplore is the number of new exploration samples added in each round of retraining; good optimization performance has been found with nexplore = nexploit; smaller values of nnew will lead to more frequent retraining and tolerance re-estimation during the optimization, and larger values of nnew will have less frequent retraining; the strategy utilized for exploitation sampling involves a series of surrogate-assisted optimizations; for each performance objective (not including tolerance), a single-objective optimization will be carried out to optimize each individual objective using the surrogate models; here, the Covariance Matrix Adaptation Evolutionary Strategy is implemented as a robust global search algorithm with a population size of 100 and a total number of function evaluations set to 300p2; these single-objective designs are then used to seed a multi-objective optimization, which then optimizes the performance objectives in the cost function as well as the tolerance hypervolume described in Section III; seeding a multi-objective optimization with designs that are estimated to be optimal in each objective is a proven technique for enhancing the performance of a multi-objective optimizer; if available, previous Pareto set designs can be included as additional seeds to the multi-objective optimization; it should be noted that the tolerance computations occur outside of the cost function and effectively add one output to the cost function, such that T + 1 output objectives will exist; this additional computation is handled internally within MOTOL (multi-objective optimization with tolerance analysis) and adds to the black-box nature of the algorithm; the cost function utilized for the tolerance hypervolume objective is provided in the equation (5), such that a minimized cost leads to a maximized tolerance hypervolume; the resulting Pareto set yields a suite of candidate designs that are estimated to be Pareto optimal by the surrogate models; these high-performing candidate designs are ranked, such that those located furthest from existing Pareto optimal designs while still being nondominated in objective space are the most desirable candidates; this ensures that a diverse and evenly sampled Pareto front is obtained; the top nexploit ranked samples are then validated with the expensive cost function; if the stopping criteria (such as a maximum number of expensive function evaluations) for the optimization have not been met, then nexplore designs are sampled with the cost function via LHS and the process iterates until the stopping criteria is met).
RAMANATH in view of AHLFELD and KAY, and Easum are analogous art because they are from the same field of endeavor, a system and a method relating to optimization with tolerance/uncertainty analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Easum to RAMANATH in view of AHLFELD and KAY. Motivation for doing so would produce designs that are robust with respect to input parameter tolerances/uncertainty (.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yang et al. ("Towards Reliable Uncertainty Quantification via Deep Ensembles in Multi-output Regression Task", arXiv:2303.16210v4, Nov. 24, 2023, pp. 1-30) discloses in Abstract and Section 1 with FIG. 1 of Pages 1-4 that (1) propose the deep ensemble framework that applies the post-hoc calibration method to improve its uncertainty quantification performance; (2) compared with Gaussian process regression and is shown to have superior performance in terms of regression accuracy (↑ 55 ∼ 56%), reliability of estimated uncertainty (↑ 38 ∼ 77%), and training efficiency (↑ 78%); (3) finally, the potential impact of the suggested framework on the Bayesian optimization is briefly examined, indicating that deep ensemble without calibration may lead to unintended exploratory behavior; (4) this UQ framework can be seamlessly applied and extended to any regression task, as no special assumptions have been made for the specific problem used in this study; (5) in the decision-making process based on the regression model, engineers should consider the predictive uncertainty derived from insufficient train data and imperfect regression model; (2) otherwise, blind faith in regression models, especially during risk assessment and management procedures, can lead to unexpected and therefore disastrous outcomes; (6) the most common approach to deal with this issue is to perform Bayesian optimization, also known as efficient global optimization in engineering fields; (7) briefly, it aims to reduce model uncertainty by iteratively updating the model based on the acquisition function, which contains uncertainty information (Fig. 1); (8) since the Bayesian optimization process requires uncertainty quantification (UQ), whether the model quantifies the uncertainty over its prediction is the key consideration for engineers in determining which regression model to utilize; (9) Gaussian process regression (GPR)—also known as Kriging—is one of the most widely used regression models capable of UQ in various engineering fields; (10) GPR allows engineers to identify which predictions are unreliable by providing predictive uncertainty, and it has become the most prevalent regression model for Bayesian optimization; (11) however, GPR is notorious for its time complexity of O(n3) and memory complexity of O(n2), where n denotes the dataset size; (12) deep ensemble (DE), an approach to quantify the predictive uncertainty by leveraging ensembles of NNs, was first proposed by Lakshminarayanan et al. (2017); (13) their idea is so “simple and straightforward” that it only requires training multiple NNs in parallel on the same training dataset; (14) despite its simplicity, several researchers have recognized that the DE provides not only accurate predictions, but also robust, reliable, and practically useful uncertainty on a wide variety of architectures and datasets, even on out-of-distribution (OOD) examples; (15) our research focuses on a thorough validation of the DE approach in multi-output regression tasks, while comparing it with GPR, both in terms of regression accuracy and reliability of the estimated uncertainty; (16) finally, a tendency of the quantified uncertainty to become underconfident with the number of NNs is observed and a practical calibration method is proposed to be applied; (17) the corresponding effects are verified quantitatively with two uncertainty evaluation criteria, and their potential impact on Bayesian optimization is briefly investigated; and (18) the main contributions of this work can be summarized as follows: (a) first attempt to validate DE approach in the multi-output regression task; (b) the effect of the number of NNs used for DE is comprehensively investigated and two different criteria are utilized for rigorous validation of its uncertainty quality; (c) accordingly, an increasing trend of under-confidence with the increasing number of NNs is first empirically observed in the regression task, and its analytical explanation is derived; (d) a simple post-hoc calibration method is applied to DE models for the correction of unsatisfactory uncertainty quality and its effectiveness is verified both qualitatively and quantitatively; (e) the potential impact of the proposed calibration method on Bayesian optimization is briefly examined: the possibility that different estimates of uncertainty could lead to different exploration behavior is examined; and (f) throughout the above procedures, GPR—the most well-known UQ model—is compared with DE, and the effectiveness of DE over GPR is confirmed. Yang further discloses in Section 2 with Algorithms 1-4 and FIGS. 2-4 of Pages 5-12 that (1) DE is based on an ensemble of NNs, but there is a key distinction: unlike a standard NN, which only outputs QoIs as μ(x), the NN used for DE outputs them as a Gaussian distribution, N(μ(x), σ2(x)); i.e., it assumes that QoIs are sampled from N(μ(x), σ2(x)) and aims to provide information about this distribution by outputting μ(x) and σ2(x), where μ(x) refers to the estimated/predicted value and σ2(x) refers to the estimated/predicted variance; (2) it should be noted that the estimated variance σ2(x) indicates the aleatory uncertainty (uncertainty arising from noise inherent in the training data) regarding the estimated value μ(x); (3) due to the probabilistic distribution it provides, this type of NN is referred to as a probabilistic NN; (4) using a single probabilistic NN is limited to estimating the aleatory uncertainty; (5) to estimate the epistemic uncertainty arising from the model parameters due to insufficient training data, a further step is required; (6) Lakshminarayanan et al. (2017) suggested the use of multiple probabilistic NNs, called deep ensemble (DE), to quantify both aleatory and epistemic uncertainties; (7) specifically, they aimed to capture the epistemic uncertainty by using the multiple probabilistic NNs trained on the identical dataset (also identical architectures for NNs are used), where the overall training procedure is summarized in Algorithm 1; (8) there are two notable points herein: a) the random initialization of the model parameters of the NNs in line 2; and b) the random shuffling of the training dataset due to mini-batches in line 5; (9) these two factors are regarded as the main causes of the individual NN with identical architecture in the ensemble being able to be trained with enough diversity; (10) Lakshminarayanan et al. (2017) suggested approximating the final probability of the output as a mixture of Gaussian probabilities as Equations (5)-(6); (11) the overall flowchart of DE from the training of probabilistic NNs to the final prediction is schematically shown in Fig. 2; (12) engineers who are interested in predicting uncertainty require more than just the feasibility of UQ, and they also require confidence in the reliability of the estimated uncertainty; (13) the most widely used metric to evaluate the reliability of uncertainty is the area under the calibration error curve (AUCE); (14) the primary goal of this measure is to ensure that the confidence intervals (CI) estimated by the model are accurate in practice; (15) the concept of AUCE is shown schematically in Fig. 3; (16) in Fig. 3a, the CI labeled “Well-calibrated 60% CI” contains 60% of the test dataset (6 out of 10 points), where test dataset indicates the dataset used to verify the quality of the estimated uncertainty; (17) thus, a well-calibrated model would have a 60% CI that actually contains 60% of the test data; (18) on the other hand, if the 60% CI contains more than 60% of the dataset (8 out of 10 points), the model is considered underconfident, which corresponds to the case of “Underconfident 60% CI”, which means that the model is not confident enough about its prediction and overestimates its CI; (19) conversely, if the 60% CI contains less than 60% of the dataset (4 out of 10 points, “Overconfident 60% CI” case), the model is considered overconfident, meaning that it is too confident in its prediction and thus estimates a narrower CI than it actually should; (20) the difference between the CI estimated by the model and the actual data it contains can be assessed visually by the CI-based reliability plot shown in Fig. 3b, where this plot compares the predicted CI from the model on the x-axis with the observed CI measured with the test dataset on the y-axis; (21) in this context, the line y = x represents an ideally well-calibrated model where the predicted CI perfectly matches the observed CI; (22) the algorithm for the CI-based reliability plot is summarized in Algorithm 2; (23) by utilizing this CI-based reliability plot, the AUCE, which is a metric that evaluates the quality of the estimated uncertainty, can be derived; (24) in detail, it is calculated as the area between the ideal line y = x and the reliability plot of the model; (25) the hatched area in Fig. 3b corresponds to the AUCE of the underconfident model, and the mathematical expression for the AUCE is provided in the equation (7), where K refers to the number of CI candidates as in Algorithm 2; (26) by definition, a low AUCE value implies that the predictive uncertainty quantified by the model is reliable (or well-calibrated); (27) despite its reputation as a metric of uncertainty quality, AUCE has a critical shortcoming in that it only considers the average over the entire test dataset rather than individuals; (28) moreover, they analytically and empirically elaborated that AUCE can be zero even when the predicted distribution is statistically independent from that of the ground truth; (29) in this context, Levi et al. (2022) proposed a novel approach to evaluate the quality of uncertainty, the expected normalized calibration error (ENCE); (30) it was first proposed based on the intuitive assumption: for the well-calibrated model, the estimated uncertainty σ2(x) will be equal to (y−μ(x))2, MSE; (31) this condition can be expressed mathematically as in Eq. 8, implying that a higher estimated variance should correspond to a higher expected MSE; (32) the Eq. 8 indicates that the ideally (perfectly) well-calibrated model will have an expected error exactly equal to predictive uncertainty; (33) in this sense, whether the model is well-calibrated can be visually inspected using the error-based reliability plot (Fig. 4), wherein y = x line indicates the ideally calibrated model; (34) the procedure for its plotting is summarized in Algorithm 3; (34) then, the area between the ideal y = x line and the error-based reliability plot can be calculated; (35) the normalized version of this value refers to ENCE, the second uncertainty quality metric, and is as Eq. 9, where B indicates the number of bins in Algorithm 3; (36) therefore, the ENCE of the underconfident model in Fig. 4 can be calculated as the hatched area divided by RMV; (37) as with AUCE, the lower the ENCE value, the better the model is calibrated; (38) in situations where the estimated uncertainty from the model is imprecise in terms of AUCE and ENCE, there are various techniques for calibrating uncertainty; (39) given the practicality being a crucial consideration in applying UQ techniques to the engineering domain, this research adopts a straightforward approach: temperature scaling; (40) more specifically, the study employs STD scaling, which is a regression task version of temperature scaling; (41) with STD scaling, it is only necessary to determine a scalar parameter, denoted as s, which is used to multiply the standard deviation initially estimated by the DE model,
σ
^
; (42) the value of s used in the calibration process is selected to minimize the NLL, as shown in Eq. 10; (43) please note that this equation is the simple modification of Eq. 4, where σ(x) is replaced by s
σ
^
(x); (44) this calibration procedure is completely separate from the training procedure of DE, and it is performed after the mixture step in Fig. 2, so it is called the post-hoc or post-process calibration method; (45) it is important to note that this calibration process is intended solely to correct the estimated uncertainty, and therefore, only the output
σ
^
(x) of the DE changes, while the predictive value
μ
^
(x) remains unaltered; (46) the steps involved in the STD calibration process are outlined in Algorithm 4; (47) for calibration, a separate dataset should be used that is distinct from the training and test datasets to ensure calibration generalization, and therefore, a validation dataset is utilized for the calibration; (48) in multi-output regression tasks, every DE output can be calibrated independently using the number of scaling parameters s equal to the output dimension (this is implemented by the for-loop in line 3 of Algorithm 4); and (49) in conclusion, this study uses a straightforward STD calibration method for uncertainty calibration, which involves tuning scalar parameters without modifying trained NNs. Yang also discloses in Section 4 with FIGS. 9-12 and Table 1 of Pages 16-19 that (1) proposed to apply the STD calibration method to the trained DE models; (2) STD calibration is simple and practical in that it requires only a single for-loop, leveraging the already trained models without additional training; (3) since the scaling factor of 1 corresponds to the case without calibration, the candidates s are set around 1; (4) accordingly, s = 10x are chosen as candidates, where x are 100 uniformly distributed points from -2 to 0.18, so that the resulting range of scaling factors to explore is from 0.01 to 1.5; (5) note that with s less than 1, underconfident models that overestimate the standard deviations (uncertainty) can be calibrated; (6) finally, the STD calibration is performed using calibration dataset split, and the optimized scaling factors for each DE model with respect to each output (QoI) are summarized in Table 1; (7) since the scaling factors are optimized to have values less than 1 during the STD calibration process (Table 1), it is obvious that the overall predictive uncertainty of DE models would decrease; (8) to provide a more intuitive understanding of the practical implications of calibration, this section extends DE models to Bayesian optimization; (9) specially, the importance of calibration in DE is highlighted by contrasting the next query candidates obtained from Bayesian optimization before and after STD calibration; (10) a test problem is adopted to practically investigate their effects on Bayesian optimization; (11) first, the multi-objective Bayesian optimization problem is defined: the maximization of both CNF and CAF within five varying input parameters (Ma, ϕ, δp, δr, and AoA); (12) these optimizations, coupled with the expected improvement (EI) acquisition function, are performed separately for DE-bef and DE-aft models; (12) the former searches for the maximum EI point where EI is calculated from the uncertainty quantified by the DE-bef model, while the latter does so using the uncertainty quantified by DE-aft; (13) to provide a brief insight into the impact of STD calibration on Bayesian optimization, only the first iteration is executed; (14) to find the Pareto solutions of EI(CNF) and EI(CAF), the non-dominated sorting genetic algorithm-II (NSGA-II) in the Python package pymoo is utilized; (15) finally, the obtained Pareto solutions from the first iteration are shown in Fig. 12a; (16) since the uncertainty estimated by DE-bef and DE-aft are different as shown in Fig. 11, the Pareto solutions of EI(CNF) and EI(CAF) are also different: EI values of both QoIs after calibration are much smaller than those before calibration; (17) in Bayesian optimization, however, the most valuable information to the user is not the EI value itself (Fig. 12a); (18) more important are the values of the input variable sets (Fig. 12b) obtained from the EI Pareto solutions, since they are the next query candidates, the main purpose of implementing Bayesian optimization; (19) if inappropriate candidates are obtained due to inaccurate UQ and therefore inaccurate EI calculation, the convergence of Bayesian optimization will be severely degraded; (20) in this sense, the parallel coordinates plot (PCP) in Fig. 12b shows how the first query candidates in Bayesian optimization can vary due to the STD calibration in the DE model; (21) each red/blue line represents each point of the Pareto solutions in Fig. 12a; (22) comparing them, large variations are found especially in the input variable δr; i.e., Bayesian optimization coupled with DE-bef discourages exploration of the variable δr, while DE-aft encourages exploration within δr; (23) as it can be regarded that the input variable sets of the next query from DE-aft are more accurate, the incorrect exploration trend of DE-bef will have significant side effects on the efficiency of the Bayesian optimization process; and (24) in conclusion, whether the DE is calibrated by STD calibration or not can result in completely different exploration characteristics when extended to Bayesian optimization.
Gel et al. ("Comparison of Deterministic and Bayesian Calibration of MFiX-PIC, Part 1: Settling Bed", arXiv:2305.01132v1, May 2, 2023, pp. 1-33) discloses in Section 3.1 of Pages 3-4 that (1) calibrating input parameters for computational simulation first requires a user to define quantities of interest/response variables, and these are measurable values that universally define the accuracy of a simulation; (2) there may be many input parameters that affect these quantities of interest, and the effect of changing those parameters may be interrelated; e.g., calibrating five input parameters for a single response variable might require thousands of evaluations to find an optimal set of parameters that yield the smallest residual between a simulated and experimental quantity of interest; (3) to avoid running these thousands of simulations, it is common to construct a surrogate model (a.k.a. a response surface or meta-model) and use it to predict simulation outcomes instead; (4) in this study, a data-fitted surrogate model, which characterizes the relationship between a response variable and input parameters through sampling simulations that span user prescribed ranges of input parameters was created; (5) in this work, the language simulation campaign describes carefully designed samples of simulations, chosen to create a numerical relationship between input parameters and a response variable; (6) intuitively it seems the number of sampling simulations in a simulation campaign must play a critical role in constructing a reliable data-fitted surrogate model; (7) in this study, a particular Optimal Latin Hypercube (OLH) sampling method is employed whereby a distance metric effectively distributes input parameters to fully span user-defined ranges while ensuring samples are located far from each other; (8) the workflow outlined below was followed to design the simulation campaign and to construct the data-fitted surrogate models: (a) identify the model input parameters to be varied systematically as part of the sampling simulations, and the quantities of interest to be extracted from the results; (b) design the simulation campaign employing OLH sampling principles; (c) launch and monitor the simulation campaign on the targeted HPC system; (d) post-process the results from simulations to construct a tabular dataset where each row shows the six model parameter settings and simulation results for each of the quantities of interest corresponding to that sampling simulation; and (e) import the tabulated dataset into the UQ toolkit software employed, and test different surrogate model options to determine the best data-fitted surrogate model for the given dataset using various statistical metrics; and (8) once a best data-fitted surrogate model was identified, this same surrogate model was used throughout the subsequent calibration process in lieu of further MFiX-PIC simulations. Gel further discloses in Section 3.2 of Page 4 with FIG. 13 of Page 17 that (1) sensitivity analysis is one UQ technique employed to address the important question: “which input parameters have the most influence on a quantity of interest?"; (2) for calibration purposes, sensitivity analysis plays a key role, particularly when the number of input parameters exceeds three; (3) the technique quantitatively determines the most influential parameters for each quantity of interest, and can be used to focus the attention of experimentalists, particularly when resources are limited; (4) in the current study, sensitivity analysis identified two key model input parameters in addition to a design variable, which was not targeted for calibration; (5) the sensitivity analysis results shown later in this report (Figure 13) were obtained using the Sobol’ Indices-based global sensitivity method, which is the preferred methodology for cases with non-linear response behavior; and (6) the data-fitted surrogate model was used to perform function evaluations for computing the quantity of interest when calculating the Sobol’ indices. Gel also discloses in Section 3.3 with FIG. 2 of Page 5 that (1) computational models often incorporate empirical input parameters as well as physically observable input parameters; (2) the intent of calibration is to tune input parameters with the aid of observable data (e.g., experiments) so that a computational model reproduces expected physics in simulations; (3) Figure 2 shows a simple sketch to illustrate the objective of calibration (Adams et al., 2015); (3) the calibration process aims to minimize the difference between the target and simulation output by adjusting the settings for the θ parameters; (4) validation is direct comparison of simulation results to experimental results without tuning; (5) one might use validation to establish a baseline discrepancy between an experiment and a simulation, and use that information to justify the need for model calibration; (6) both validation and calibration are always performed against a specific set of observable data, which makes the credibility of the experimental data quite critical; (7) in general, calibration methods are categorized under two groups: (i) deterministic calibration methods, and (ii) statistical calibration methods; (8) the latter provides a distribution for the calibrated model parameters instead of single values, which is the outcome of deterministic calibration; and (9) another major difference is the ability of statistical calibration to take into account model bias (a.k.a. model form uncertainty) while performing calibration of model input parameters. Gel further teaches in Section 3.4 with FIG. 3 in Pages 5-7 that (1) the goal of deterministic calibration is to find values of θ : {θ1, … θm} that will minimize residual error between a group of simulations and their equivalent experimental counterparts; (2) Eqn. 5 acts as the objective function for the optimization problem (Adams et al., 2015), which represents the sum of squares of the residual errors introduced by employing this set of θ in n simulations; (3) an important distinction between statistical calibration and deterministic calibration is that the outcome from statistical calibration is an estimated distribution of the θ parameters individually, whereas deterministic calibration provides a single scalar value for each of the model parameters being calibrated; and (4) the workflow outlined below was followed to perform deterministic calibration in this study: (a) identify the model parameters to be calibrated, and determine the lower and upper bounds for each of these parameters to be used during calibration; (b) prepare an experimental dataset or observations to be used to guide the calibration process as an ASCII input file; (c) plan a simulation campaign with the aid of statistical design of experiments principles that will enable the construction of a data-fitted surrogate model, wherein the surrogate model should adequately characterize the relationship between model parameters considered as input and the response variables (a.k.a. quantities of interest or output), and this step is crucial when the simulations are expensive or time consuming to perform as the optimization process requires thousands of function evaluations to be performed at a low computational cost; (d) post-process the simulation campaign results and compile an ASCII file as a tabulated dataset consisting of the design of experiments for the model parameters and the corresponding quantities of interest from the simulation campaign results; (e) utilize UQ toolkit (PSUADE, Nodeworks) to import the datasets and perform the optimization required to minimize the residuals in Eqn. 5; and (f) verify the proposed calibrated model parameter settings by re-running a select group of simulations within the existing simulation campaign or by constructing a new simulation campaign for unseen samples. Gel also teaches in Section 3.5 with FIG. 4 of Pages 7-9 that (1) Bayesian calibration employs Bayes Theorem, which simply relates prior information with associated uncertainty to future information based on the likelihood of observed outputs from the model (Muehleisen and Bergerson, 2016); (2) in the deterministic calibration method, the objective is to find a set of values for the uncertain model parameters that minimize the residual error difference between observed data from experiments and model computed quantities of interest; (3) however, in Bayesian calibration, the objective is to determine the most likely uncertainties for input parameters that yield the quantity of interest, with some uncertainty, in which the observed data is most likely to reside (Muehleisen and Bergerson, 2016), which is an iterative process of updating distributions with targeted uncertain parameters in a way that is consistent with observed data; (4) Figure 4 shows a high-level illustration of the Bayesian calibration framework, which starts with a prior distribution of the model parameters based on current beliefs; (5) hence, the assumptions for prior distribution makes a difference; (6) then observations from the experiments are employed to guide the calibration process which simply employs Bayes Theorem to estimate the posterior distribution of the model parameters; (7) the Markov Chain Monte Carlo (MCMC) based approach is employed to perform Bayes’s rule; (8) in this study, the original Bayesian calibration framework from Kennedy and O’Hagan (2001) was followed, which is based on representing model bias and quantities of interest from the computer model as Gaussian processes, to investigate how the sophisticated tuning of the five MFiX-PIC model parameters can improve the prediction accuracy of the location of the settling shock; (9) a major deviation from the original Bayesian calibration framework was the availability of the observations to guide the calibration; (10) typically, experimental data with some uncertainty is utilized whereas an analytical solution was employed, and artificial uncertainty (less than 1%) was introduced to characterize the effect of experimental uncertainty; (11) Bayesian calibration starts with a prior distribution of the uncertain model parameters, which reflects our beliefs about the parameters; (12) then Bayes Theorem is employed to update initial beliefs in the model parameters with the help of observations; (13) hence, the posterior distribution of the model parameters can be calculated as the solution of Eqn. 7 for a given set of observations and simulator results; (14) besides estimate uncertainties in the calibration parameters θ, one can also perform predictions at points that were not previously observed (simulations or experiments) using the Gaussian Process models constructed; (15) following the outline in Muehleisen and Bergerson (2016), the general workflow for Bayesian calibration can be described under three major steps: (a) define prior distributions based on the beliefs about uncertain model parameters (e.g., characterize the uncertainty for θ1 to θ5 with assumed PDFs, which for this case was uniform distribution within a prescribed lower and upper bounds); (b) collect experimental observations based on the design variables (e.g., for this case x1: initial concentration was varied to compute the location of settling shock from the analytical solution, which was used in lieu of the experiments); (c) calibrate (assumed) prior parameter PDFs based on the observed data by iteratively using Bayes’ Theorem until iterations converge to an acceptable level (Kennedy and O’Hagan, 2001), which yields the estimated posterior distribution of the model parameters considered for calibration (i.e., θ1 to θ5); and (16) due to the computational complexity and intensive resource requirements of the above workflow, an open-source UQ toolkit which can partially automate the above steps has been employed in this study. Gel further discloses in Section 3.5 of Pages 9-10 that (1) PSUADE is an open-source UQ software toolkit developed at the Lawrence Livermore National Laboratory (Tong, 2010) and released under LGPL license since 2007, wherein the name of the software, PSUADE, comes from the acronym for Problem Solving Environment for Uncertainty Analysis and Design Exploration; (2) the program supports a variety of non-intrusive uncertainty quantification analysis methods where the simulation application can be treated as "black-box" code; (3) subsequently, many UQ analysis tasks can be performed by sampling the black-box directly or through a data-fitted surrogate model constructed from the computational model; (4) the software offers a diverse range of sampling methods to enable users to perform simulation campaigns with the objective of constructing an adequate data-fitted surrogate model (a.k.a. response surface model, meta-model); (5) the user can perform both basic uncertainty analysis such as forward propagation of uncertainties and more complex analysis such as mixed aleatory-epistemic uncertainty analysis; and (6) PSUADE has a built-in statistical calibration capability (i.e., Bayesian calibration with MCMC).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142
Prosecution Insights