Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 219 (Fig. 2, "CONNECTION DATA"), 245 (Fig. 2, "MEMORY CONFIGURATION VALIDATOR"). Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities: (¶[0017] contains the misspelling "handing," which should read "handling," in the phrase "a read disturb handing (RDH) window"; (2) ¶[0075] incorrectly refers to "the operation 350" where the context and FIG. 3 indicate this should read "the operation 330"; (3) the specification uses inconsistent titles — "MACHINE LEARNING-BASED ADJUSTMENT OF MEMORY CONFIGURATION PARAMETERS" on the cover page versus "MACHINE LEARNING BASED MEMORY CONFIGURATION PARAMETER CONSTRUCTION IN A MEMORY DEVICE" in the body of the specification. Appropriate correction is required.
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required: Claims 9, 15, and 20 recite "a read disturb handling (RDH) threshold," but the specification describes only an "RDH window" (¶[0017], [0082]) and does not use or describe an "RDH threshold." Claims 9, 15, and 20 further recite "a read level offset threshold," but the specification describes only "a read level offset" (¶[0017], [0026], [0062], [0082]) and does not use or describe a "read level offset threshold."
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Written Description
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Per claims 1-9 and 16-20, there is no described mechanism for determining that the predicted values satisfy a confidence criterion.
Claim 1 recites “responsive to determining that the set of predicted values of the memory configuration parameters satisfies a confidence criterion, updating the memory configuration parameters to reflect the set of predicted values”. Claim 16 recites the same limitation. Claim 2 further recites “updating a value of a confidence indicator, wherein the value of the confidence indicator reflects whether the confidence criterion was satisfied”, and claim 17 recites the same.
The specification does not describe how the recited determination is made, that is, how a confidence value is derived for a given set of predicted values of the memory configuration parameters and evaluated against the criterion. What the disclosure provides is a statement of what the criterion signifies and where it originates, not a description of the determination itself. At ¶[0062]: “Output 207 can also include an indication of whether the predicted values of memory configuration parameters 211 satisfy confidence criteria 217.” This states that the indication exists; it does not describe how the indication is produced. At ¶[0064]: “Confidence criteria 217 can include threshold values for predicting predicted values of memory configuration parameters 211 based on values of performance metric 212.” This statement is circular and does not identify the quantity to which any threshold is applied. At ¶[0074]: the confidence criterion “can reflect an estimated likelihood that the memory access operation based on the predicted values of the memory configuration parameters will satisfy the threshold condition,” the determination “can be a binary (i.e., yes/no) value,” and the criterion “can be based on an importance assigned to the memory access operation.” No method of estimating the likelihood is given. At ¶[0087]: “Confidence criterion and calculations to determine the confidence indicator with respect to the confidence criterion can be set by a production system, and can depend on manufacturing techniques, design requirements, memory device characteristics, etc.” The disclosure thus expressly defers the dispositive calculations to an undescribed production system, and no such calculation appears anywhere in the specification.
The only confidence quantity actually described in the disclosure is the decision-tree “branch confidence score” at ¶[0060], which is a per-branch classification confidence attached to a leaf node and is not described as being computed for, or applied to, a set of predicted values of memory configuration parameters. No confidence computation is described for the linear or SVM regression models of ¶[0056], whose Formulas 1 and 2 return only a fitted value; for the artificial neural network of ¶¶[0057]-[0058]; or for the rule engine of ¶[0061]. Because claims 1 and 16 are not limited to any model species, the specification fails to convey possession of the claimed determination across the scope claimed. See MPEP § 2163; MPEP § 2161.01(I); Vasudevan, 782 F.3d at 682-83.
Claim 2 and claim 17 compound rather than cure the deficiency: ¶[0087] describes generating a confidence indicator, not updating a value of a pre-existing confidence indicator, and it is that same paragraph that defers the underlying calculations.
Claims 2-9 and 17-20 are rejected as depending from claims 1 and 16 respectively and incorporating the same deficiency.
Per claims 6 and 7, recites decision tree and rule engine limitations with no counterpart in the disclosure.
Claim 6 recites a decision tree model “wherein a first branch set of branches comprises the threshold condition, wherein a second branch of the set of branches comprises the confidence criterion, wherein a first leaf of the decision tree model comprises the output from the trainable model and a branch path”. The specification's decision-tree disclosure at ¶¶[0059]-[0060] is generic: branch nodes carry tests, leaf nodes carry outcomes and branch path data, and the sole illustration is an abstract feature-vector test…for X = [0,-1,-1,-1,-1], “Is the first element of vector X greater than or equal to zero?” with the outcome “predicted as ‘taken’.” The specification nowhere describes a decision tree in which a branch comprises the threshold condition of the memory access operation, nor one in which a branch comprises the confidence criterion, nor one whose leaves comprise predicted values of memory configuration parameters. The one confidence quantity that is described is placed at a leaf (¶[0060]), not at a branch test as claim 6 requires.
Claim 7 recites a trainable model as a rule engine “wherein a first rule of the set of rules comprises the threshold condition, wherein a second rule of the set of rules comprises the confidence criterion, wherein a first action of a set of actions comprises determining the value of the performance metric”. The specification's rule-engine disclosure at ¶[0061] is likewise generic — “a selection rule can define a logical condition and an action to be performed if the logical condition is evaluated as true” — and describes rules as “predefined and/or dynamically configurable,” not as trained. No rule comprising the threshold condition, no rule comprising the confidence criterion, and no action that determines the value of the performance metric is described. The last limitation is additionally at odds with the claim from which claim 7 depends: claim 1 recites providing the value of the performance metric as an input to the trainable model, whereas claim 7 requires the trainable model itself to determine that value. The specification describes the performance metric value as originating in device design requirements and measurement (¶[0072]; ¶[0081]), never as an action or output of the model.
The specification further describes no technique for training either a decision tree model or a rule engine. The training techniques identified are limited to “a linear regression training technique, an SVM regression training technique, a neural network training technique” (¶[0094]); see also ¶[0031] and ¶¶[0052]-[0053], which describe error minimization and the adjustment of node weights. The recitation at page ¶[0055] that “configuration model 202 can be a regression model, a neural network, a decision tree, and/or a rule engine” is a bare listing; naming a species does not describe how that species is trained or how the claimed content mapping is realized within it. See MPEP § 2163.
Per claims 9, 15 and 20, the recited memory configuration parameters are not described.
Claims 9, 15 and 20 each recite that the memory configuration parameters “comprise at least one of” an enumerated list that includes “a read level offset threshold” and “a read disturb handling (RDH) threshold”. The specification's corresponding enumeration at ¶[0017] recites “a read level offset” and “a read disturb handing (RDH) window”, not a read level offset threshold and not an RDH threshold. The supporting text is to the same effect: ¶[0026] (“Read level offsets are used for memory access operations on a set of cells for a voltage distribution that has shifted from an initial programming position”) and ¶[0027] (“the window size of the RDH operation can determine the speed and accuracy of the RDH operation”). The later enumerations at ¶[0082] and ¶[0098] likewise recite “a read level offset” and “a RDH window.”
Because each of claims 9, 15 and 20 uses the open “comprise at least one of” format, each claim reads on an embodiment in which the memory configuration parameters comprise only a read level offset threshold, or only a read disturb handling (RDH) threshold. The specification describes neither parameter and therefore does not convey possession of the full scope claimed. Satisfaction of the limitation by one of the described alternatives does not cure the unsupported alternatives. See MPEP § 2163.
Per claims 10-15, the recited training data is inverted with respect to the recited prediction. Claim 10 recites “generating, by a processing device, training data for training a trainable model to predict a predicted set of values of memory configuration parameters based on a value of a performance metric”. The claim then specifies the training data: the training input “compris[es] a historical set of values of the memory configuration parameters for a prior memory device”, and the target output “comprises a historical value of the performance metric ... and an indication of whether the threshold condition is satisfied”.
The recited training data therefore places memory configuration parameter values on the input side and performance metric values on the target side. Supervised training on that pairing, as the specification describes training, produces a model that maps memory configuration parameter values to a performance metric value. That is the inverse of the prediction the claim requires the model to be trained to perform, namely predicting a set of memory configuration parameter values from a value of a performance metric. The specification does not describe how a model trained on the recited training data performs the recited prediction.
The only training procedures described are error-minimizing supervised procedures that fit the model output to the target output. See specification ¶[0052] (“feeding a training dataset consisting of labeled inputs 203 through configuration model 202, observing the model outputs 207, defining an error (by measuring the difference between the outputs and the label values), and adjusting parameters of configuration model 202 to minimize the error”); ¶[0053]; and ¶[0094] (“a supervised training technique such as a linear regression training technique, an SVM regression training technique, a neural network training technique”). A model so trained produces target-domain values from input-domain values.
The specification's description of FIG. 5 confirms the same inverted arrangement and then simply asserts the desired result. At ¶[0091], the processing device “generates quasi-random values of memory configuration parameters for model inputs”; at ¶[0092] it “obtains drive level measurements based on the quasi-random values of the memory configuration parameters to derive model outputs”; at ¶[0094] it “builds the trainable model using generated trainable inputs and obtained target outputs”; and at ¶[0095], “based on a target output requirement, the processing device determines values of the memory configuration parameters for a memory device.” The specification at ¶[0095] states the result to be achieved but describes no procedure for achieving it. There is no inversion of the fitted model, no search or optimization over the memory configuration parameter space, no objective function, no convergence or selection criterion, and no treatment of the fact that different sets of memory configuration parameter values may yield the same performance metric value, which makes the inverse mapping ill-posed on the face of the disclosure.
The specification's two statements on the orientation of the training data are inconsistent with one another and are never reconciled. At ¶[0031], “historical values of memory configuration parameters can be provided as a training input for the trainable model, with historical values of performance metrics as a target output,” which matches claim 10. At ¶[0063], “historical performance metric values 216 can be used as input 203 to generate output 207,” which “can include predicted values of the memory configuration parameters 211” — the opposite arrangement. The disclosure nowhere explains how the arrangement recited in claim 10 yields the prediction recited in claim 10.
That claim 10 recites language also appearing in the specification at ¶¶[0098]-[0101] does not cure the deficiency. Those paragraphs restate the claim and add no mechanism. Written description is not satisfied where the disclosure sets out only the result to be obtained; “a mere wish or plan for obtaining the claimed invention” is not an adequate written description. Ariad Pharms., Inc. v. Eli Lilly & Co., 598 F.3d 1336, 1349-50 (Fed. Cir. 2010) (en banc); Regents of the Univ. of Cal. v. Eli Lilly & Co., 119 F.3d 1559, 1566 (Fed. Cir. 1997); MPEP § 2163. For computer-implemented subject matter, the specification must describe the manner in which the claimed function is performed, not merely that it is performed. Vasudevan Software, Inc. v. MicroStrategy, Inc., 782 F.3d 671, 682-83 (Fed. Cir. 2015); MPEP § 2161.01(I).
Claims 11-15 are rejected as depending from claim 10 and incorporating the same deficiency. None of claims 11-15 supplies the missing description; claim 14 confirms that the deficiency extends across all four disclosed model species, and claim 15 across all enumerated parameters.
Enablement
Claims 7 and 10-15 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
To satisfy the enablement requirement of 35 U.S.C. § 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, the specification must teach those skilled in the art how to make and use the full scope of the claimed invention without “undue experimentation” (see MPEP § 2161.01(III)). The factors to be considered in determining whether a disclosure meets that requirement are set forth in In re Wands, 858 F.2d 731, 737 (Fed. Cir. 1988). See MPEP § 2164.01(a).
Claim 7 requires the trainable model to be a rule engine in which a first rule comprises the threshold condition, a second rule comprises the confidence criterion, and a first action comprises determining the value of the performance metric. A showing of undue experimentation is given below.
(A) ‘The breadth of claims’ — any rule engine, any set of rules, and any set of actions falls within the claim, subject only to the three recited content constraints, one of which requires the model to determine the very value that claim 1 supplies to it as input.
(C) ‘The state of the prior art’ — rule engines and machine-learning models were both well-known as of the effective filing date, but the art supplies no technique for training a rule engine by the error-minimization procedure the specification describes, and no technique for a model whose action determines its own input value.
(F) ‘The amount of direction provided by the inventor’ — ¶[0061] is the entire rule-engine disclosure. It provides no rule-learning procedure, no rule representation for the threshold condition or the confidence criterion, and no action that determines a performance metric value. Elsewhere the specification identifies only regression, SVM regression and neural network training techniques (¶[0094]).
(G) ‘The existence of working examples’ — no rule engine example of any kind, working or prophetic, is provided.
(H) ‘The quantity of experimentation needed to make or use the invention based on the content of the disclosure’ — a person of ordinary skill would be required to devise a rule representation for the threshold condition and for the confidence criterion, a training procedure by which a rule engine is trained, and a resolution of the circularity by which an action of the model determines the performance metric value that claim 1 supplies as the model's input — all without any starting point in the disclosure. This would constitute undue experimentation.
The specification does not enable a person of ordinary skill in the art to make and use the full scope of claim 7 without undue experimentation.
Claims 10-15, as set out in the written description rejection above, claim 10 requires training data by which a trainable model is trained to predict memory configuration parameter values from a performance metric value, while specifying training data whose input and target sides are oriented the other way. A showing of undue experimentation is given below for that limitation, based on the factors cited in MPEP § 2164.01(a) as pertaining to In re Wands.
(A) ‘The breadth of claims’ — claim 10 covers generating training data for any trainable model. Claim 14 confirms that the model may be a regression model, a neural network, a decision tree model, or a rule engine, and claim 15 confirms that the predicted parameters may be any of at least twelve enumerated types, for any prior memory device fabricated at any production system. The full scope must be enabled, and the missing mechanism is missing for every species and every parameter type.
(B) ‘The nature of the invention’ — the invention as claimed is an inverse-problem application of supervised machine learning to memory device production. Because the recited training data is oriented from parameters to performance metric while the recited prediction runs from performance metric to parameters, obtaining the recited prediction requires inverting or searching over a fitted forward model. That is a materially different undertaking from the forward fitting the specification describes.
(C) ‘The state of the prior art’ — supervised regression and neural network training were well established as of the April 3, 2023 effective filing date, and to that extent the art is mature. The art does not, however, supply any standard, drop-in technique for recovering a set of input values from a fitted forward model. Doing so requires a deliberately chosen inversion, optimization or search formulation, and the specification supplies none.
(D) ‘The level of one of ordinary skill’ — Applicant cannot rely on the knowledge of one skilled in the art to supply information that is required to enable the novel aspect of the claimed invention when the enabling knowledge is not itself known in the art (MPEP § 2161.01(III)).
(E) ‘The level of predictability in the art’ — the memory device and software arts are generally predictable, but the recited arrangement is not. Whether a fitted forward model admits a usable inverse depends on the model class, on the training data, and on whether the mapping is one-to-one. On the face of the disclosure the mapping is many-to-one: the specification states that values of memory configuration parameters depend on trim parameters, physical structure, per-cell densities and circuit layout (¶[0028]), so different parameter sets may yield the same performance metric value. The inverse is therefore ill-posed and its behavior cannot be predicted from the disclosure.
(F) ‘The amount of direction provided by the inventor’ — no direction is provided for the dispositive step. At ¶¶[0091]-[0093] describe only generating quasi-random memory configuration parameter values as model inputs and obtaining drive level measurements as model outputs; ¶[0094] describes building the model from those inputs and targets; and ¶[0095] then asserts that parameter values are determined “based on a target output requirement” without describing any procedure. At ¶¶[0098]-[0102] restate the claim. No loss function, no search or optimization strategy, no constraint handling, and no convergence or selection criterion for the inversion appears anywhere in the disclosure.
(G) ‘The existence of working examples’ — there are none, working or prophetic. Formulas 1 and 2 at ¶[0056] are generic linear-regression and SVM-regression expressions in unnamed features x and coefficients c; they contain no memory-domain variable, no dataset, no parameter value, and no result. No accuracy figure, error measurement, or validation result of any kind is reported anywhere in the specification.
(H) ‘The quantity of experimentation needed to make or use the invention based on the content of the disclosure’ — a person of ordinary skill would be required to devise the entire mechanism by which the recited training data yields the recited prediction: selecting an inversion or optimization strategy, resolving the many-to-one mapping, defining an objective function and a convergence criterion, and validating the result across at least twelve parameter types and four model species. That is not routine implementation of a disclosed teaching; it is development of the dispositive step of the claimed invention, and it would constitute undue experimentation.
The specification does not enable a person of ordinary skill in the art to make and use the full scope of claim 10 without undue experimentation. Claims 11-15 are rejected for the same reason by virtue of their dependency from claim 10.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4, 5, 8-16, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over US Pat. Pub. No. 2020/0210831 to Zhang et al. (hereinafter Zhang) in view of US Pat. Pub. No. 2022/0026817 to Ummethala et al. (hereinafter Ummethala).
Per claim 1, Zhang discloses A system (Zhang: ¶[0067]…Zhang describes a deep neural network system 1100 for data storage optimization built from one or more processors and one or more memory devices, which constitutes the recited “system” under BRI, “The deep neural network 1100 may include one or more processors 1110 and one or more memory devices 1120 operatively coupled to the one or more processors 1110”)) comprising:
a memory (Zhang: ¶[0068]…Zhang’s memory devices 1120 hold the trained model’s hyperparameters and the read threshold values, which constitutes the recited “memory” under BRI, “In some embodiments of the disclosed technology, one or more of the memory devices 1120 may be configured to store a set of hyperparameters of a training model 1122 and read thresholds 1124”)); and
a processing device operatively coupled to the memory, the processing device to perform operations comprising (Zhang: ¶[0067]…Zhang’s processors 1110 are operatively coupled to the memory devices 1120 and carry out the receive-predict-modify sequence of FIG. 10, which constitutes the recited “processing device operatively coupled to the memory” that performs the recited operations under BRI, “The deep neural network 1100 may include one or more processors 1110 and one or more memory devices 1120 operatively coupled to the one or more processors 1110. One or more of the processors 1110 may be operable to receive a plurality of operating conditions that contribute to the read errors in the data storage devices”)):
providing, as an input to a trainable model, a value of a performance metric based on a threshold condition of a memory access operation performed on a memory device using a set of values of memory configuration parameters (Zhang: ¶[0060]…Zhang’s already-trained deep neural network receives, at its input nodes, the operating conditions that contribute to read errors - among them endurance - while the underlying data is measured on the memory device using a set of read thresholds, i.e., using a set of values of memory configuration parameters; the condition values so received therefore constitute the recited input to a trainable model under BRI, “The already-trained deep neural network 740, through the plurality of input nodes 710, the first connection layer 720, and the first connection nodes 730, receives the operating conditions that contribute to the read errors, such as endurance, retention, read disturbance, die index, block index, word line index, age of the data storage drive, and/or temperature. The deep neural network 740 measures data in the memory devices under combinations of operating conditions using a set of read thresholds”; ¶[0045]…Zhang defines endurance as the maximum number of program/erase operations the device is guaranteed to perform successfully, which is a value of a performance metric based on a threshold condition (successful completion) of a memory access operation, “The endurance of flash memories may be represented by the maximum amount of program/erase operations that the flash memories are guaranteed to be able to perform program/erase operations successfully”; ¶[0057]…Zhang further conditions resort to the network on whether the criteria for the optimal read threshold have been met, as measured by the number of errors returned by reads performed under the read thresholds then in force, confirming that the sensed quantity supplied to the engine is a threshold-conditioned performance measure, “For example, when it is determined that the criteria for the optimal read threshold is not met (e.g., when the number of errors or indication of errors from the memory device approaches an undesirably high value, a temperature value approaches a low or a high threshold value, etc.), a memory controller can obtain new values to modify the read thresholds based on the sensed operating conditions using the values that is generated by the deep learning neural network engine”));
obtaining as an output from the trainable model, a set of predicted values of the memory configuration parameters (Zhang: ¶[0066]…Zhang obtains the optimal read thresholds - a set of read threshold voltage values, which are memory configuration parameters - from the output nodes of the trained deep neural network, which constitutes the recited set of predicted values of the memory configuration parameters under BRI, “At step 1010, the current operating conditions are fed into the input nodes of the trained deep neural network. At step 1020, the optimal read thresholds are obtained from output nodes of the trained deep neural network”)); and
…updating the memory configuration parameters to reflect the set of predicted values of the memory configuration parameters (Zhang: ¶[0066]…Zhang modifies the memory device’s in-use read thresholds so that they take the values the trained network predicted, and thereafter performs reads using those modified values, which constitutes the recited updating step under BRI, “At step 1030, read thresholds are modified to the optimal read thresholds. At step 1040, data can be read out from the memory cell using the optimal read thresholds”)).
Zhang does not expressly disclose, but Ummethala does teach:
responsive to determining that the set of predicted values of the memory configuration parameters satisfies a confidence criterion, … (Ummethala: ¶[0073]…Ummethala’s trained model emits, together with each predicted value, confidence data, and the predicted value is adopted for the workpiece under test only where that level of confidence satisfies a threshold condition, which constitutes the recited confidence-criterion condition under BRI, “At block 650, processing logic uses confidence data to estimate a metrology value for the substrate being processed at the manufacturing system. In some embodiments, if the level of confidence for a metrology value satisfies a threshold condition, then a substrate is identified as being associated with the metrology value”; ¶[0035]…Ummethala further quantifies the confidence as a bounded numerical level against which the threshold condition is evaluated, “The confidence data may include or indicate a level of confidence that a metrology value corresponds one or more properties of a substrate associated with current spectral data and/or spectral data. In one example, the level of confidence is a real number between 0 and 1 inclusive, where 0 indicates no confidence that the metrology value corresponds to one or more properties of the substrate associated with the current spectral data and 1 indicates absolute confidence that the metrology value corresponds to one or more properties of the substrate associated with the current spectral data”)).
Zhang and Ummethala are analogous art because they are from the same field of endeavor, specifically the machine-learning-based prediction of operating parameter values for semiconductor devices from measured performance data. They address the same problem of arriving at parameter values that cause a manufactured device to satisfy its performance requirements without exhaustive manual measurement of every operating combination.
Before the effective filing date of the claimed invention, it would have been obvious to a PHOSITA to modify Zhang’s deep neural network engine, which computes optimized read threshold voltages and writes them back to the memory device, so that the computed values are adopted only when the model’s level of confidence in them satisfies a threshold condition as taught by Ummethala. This is the use of a known technique to improve a comparable device in the same way under MPEP 2143.01(C), yielding the predictable result of suppressing the adoption of unreliable predicted parameter values.
The suggestion/motivation for doing so would have been provided by Ummethala itself, which teaches that a predicted value is to be adopted for the workpiece under test only where the model’s confidence in that value clears a threshold, so that predictions the model is not confident in are not acted upon, “At block 650, processing logic uses confidence data to estimate a metrology value for the substrate being processed at the manufacturing system. In some embodiments, if the level of confidence for a metrology value satisfies a threshold condition, then a substrate is identified as being associated with the metrology value” (Ummethala: ¶[0073]).
Per claim 2, Zhang combined with Ummethala discloses claim 1. Zhang does not expressly disclose, but Ummethala does teach: updating a value of a confidence indicator, wherein the value of the confidence indicator reflects whether the confidence criterion was satisfied (Ummethala: ¶[0072]…Ummethala extracts a level of confidence from the model output on each prediction and carries it forward as a bounded numerical value, “At block 640, processing logic extracts confidence data from the outputs obtained at block 630. In some embodiments, the confidence data includes a level of confidence that a profile of the substrate is associated with a metrology value. In one example, the level of confidence is a real number between 0 and 1 inclusive”; ¶[0073]…that extracted value is then evaluated against the threshold condition to decide whether the predicted value is adopted, so the value it carries is what records whether the confidence criterion was satisfied, which constitutes the recited confidence indicator under BRI, “At block 650, processing logic uses confidence data to estimate a metrology value for the substrate being processed at the manufacturing system. In some embodiments, if the level of confidence for a metrology value satisfies a threshold condition, then a substrate is identified as being associated with the metrology value”)).
The rationale to combine Ummethala with Zhang is the same as the parent claim.
Per claim 4, Zhang combined with Ummethala discloses claim 1. Zhang does not expressly disclose, but Ummethala does teach: the trainable model is a regression model, wherein the regression model determines a line that matches a set of datapoints plotted along a first dimension with respect to a second dimension, wherein the first dimension is the input to the trainable model, wherein the second dimension is the output from the trainable model (Ummethala: ¶[0031]…Ummethala expressly names linear regression as an algorithm its trained model may use, and describes the model as capturing the patterns that map the training input to the target output. A linear regression model so trained necessarily fits a line to the training datapoints, and that line necessarily relates the model’s input variable to the value it predicts - that is intrinsic in the named algorithm, not merely probable, because a linear regression model has no other mechanism by which to produce an output from an input. The named algorithm therefore constitutes the recited regression model under BRI, “The training engine 182 may find patterns in the training data that map the training input to the target output (the answer to be predicted), and provide the machine learning model 190 that captures these patterns. The machine learning model 190 may use one or more of support vector machine (SVM), Radial Basis Function (RBF), clustering, supervised machine learning, semi-supervised machine learning, unsupervised machine learning, k-nearest neighbor algorithm (k-NN), linear regression, random forest, neural network (e.g., artificial neural network), etc.”)). The rationale to combine Ummethala with Zhang is the same as the parent claim.
Per claim 5, Zhang combined with Ummethala discloses claim 1. Zhang further teaches the trainable model is a neural network, wherein the output from the trainable model is generated by passing the input to the trainable model through one or more nodes of one or more layers, wherein a multivariate function represents a transformation caused by the one or more nodes of the one or more layers between the input to the trainable model and the output from the trainable model (Zhang: ¶[0061]…Zhang’s trainable model is a deep neural network whose input neurons propagate values through one or more layers of hidden neurons to the output neurons, and Zhang expresses the per-layer computation as the multivariate function shown in Eq. 1, of the weights and input activations, which constitutes the recited neural network under BRI, “The input neurons 810 receive some values and propagate them to the hidden neurons 820 of the network. The weighted sums from one or more layers of hidden neurons 820 are ultimately propagated to the output neurons 830. Here, the outputs of the neurons are often referred to as activations, and the synapses are often referred to as weights. An example of the computation at each layer can be expressed as: [Eq. 1]”)).
Per claim 8, Zhang combined with Ummethala discloses claim 1. Zhang further teaches wherein updating the memory configuration parameters comprises: updating one or more entries of a data structure accessible to a memory sub-system comprising the memory device, wherein each entry of the one or more entries of the data structure comprises one or more values of the set of values of the memory configuration parameters (Zhang: ¶[0057]…Zhang writes the values the trained network generated into a lookup table held in a memory of the data storage device that contains the memory device, and the memory controller thereafter modifies the read threshold voltages from those entries, which constitutes the recited data structure and its entries under BRI, “In another implementation, the values obtained by the trained neural network engine may be stored in a memory (e.g., a lookup table) of a data storage device, and a memory controller in the data storage device may modify the read threshold voltages based on the values”)).
Per claim 9, Zhang combined with Ummethala discloses claim 1. Zhang further teaches wherein the memory configuration parameters comprise at least one of: a folding threshold, a forgiveness threshold, a bit error rate (BER), a residual bit error rate (RBER), an error trigger rate, a read refresh rate, a quality of service (QoS) threshold, a block family error avoidance (BFEA) bin pointer, a deep-check threshold, a read level offset threshold, a read disturb handling (RDH) threshold, or a temperature compensation value (Zhang: ¶[0056]…Zhang’s memory configuration parameters are read threshold voltages - the read voltage setting applied when a read operation is conducted, “The performance (e.g., input/output operations per second and throughput) of a data storage device such as an SSD is heavily dependent on the read threshold setting (i.e., read voltage setting) applied when the first read operation is conducted. If the read threshold is not optimized, the performance may be degraded because such unoptimized read threshold voltages can cause read errors”; ¶[0055]…and Zhang modifies those read thresholds away from their nominal position to compensate the threshold-voltage shift that operating conditions produce, which is precisely the offset function the recited read level offset threshold alternative performs, “Such read errors, however, can be minimized by modifying the read thresholds. In some embodiments of the disclosed technology, the read thresholds may be modified based on operating conditions that contribute to the read errors in flash memory-based data storage SSD devices”)).
Per claim 10, Zhang discloses A method comprising:
generating, by a processing device, training data for training a trainable model to predict a predicted set of values of memory configuration parameters based on a value of a performance metric, wherein the value of the performance metric is based on a threshold condition of a memory access operation performed by a memory device using a set of values of the memory configuration parameters, wherein to generate the training data, the processing device to perform operations further comprising (Zhang: ¶[0064]…Zhang generates the labeled training data set for its deep neural network engine by reading the memory device using a first set of read threshold voltages - values of memory configuration parameters - and recording the read outcome, from which the proper read thresholds are found; the engine so trained predicts memory configuration parameter values from the threshold-conditioned performance measures Zhang senses, which constitutes the recited generating step under BRI, “At step 910, data in the memory devices are read out under certain combinations of operating conditions using a first set of read threshold voltages. For example, threshold voltages of memory cells of the memory device are measured under certain combinations of operating conditions using the first set of read threshold voltages. In an implementation, threshold voltages of memory cells are measured under certain combinations of discrete values of operating conditions using a set of read thresholds. At step 920, proper read thresholds are found to produce a labeled training data set”)): …
…and an indication of whether the threshold condition is satisfied (Zhang: ¶[0064]…Zhang’s labelling step records, for each set of read thresholds exercised on the device, whether those thresholds were proper - that is, whether the reads they produced satisfied the error criteria - and it is that pass/fail label that makes the data set a labeled one, which constitutes the recited indication of whether the threshold condition is satisfied under BRI, “At step 910, data in the memory devices are read out under certain combinations of operating conditions using a first set of read threshold voltages. For example, threshold voltages of memory cells of the memory device are measured under certain combinations of operating conditions using the first set of read threshold voltages. In an implementation, threshold voltages of memory cells are measured under certain combinations of discrete values of operating conditions using a set of read thresholds. At step 920, proper read thresholds are found to produce a labeled training data set”; ¶[0057]…Zhang states the criterion explicitly as whether the number of errors from the device has reached an undesirably high value, “For example, when it is determined that the criteria for the optimal read threshold is not met (e.g., when the number of errors or indication of errors from the memory device approaches an undesirably high value, a temperature value approaches a low or a high threshold value, etc.), a memory controller can obtain new values to modify the read thresholds based on the sensed operating conditions using the values that is generated by the deep learning neural network engine”)); and …
Zhang does not expressly disclose, but Ummethala does teach:
generating a training input comprising a historical set of values of the memory configuration parameters for a prior memory device fabricated at a production system for manufacturing the memory device (Ummethala: ¶[0004]…Ummethala builds the first training input out of historical data taken from a prior workpiece already processed at the manufacturing system that produces the workpiece under test; applying that construction to Zhang’s historical read threshold values, which Zhang collects from a large number of previously characterized dies, constitutes the recited training input under BRI, “Generating the training data includes generating a first training input including historical spectral data and/or historical non-spectral data associated with a surface of a prior substrate previously processed at the manufacturing system”));
generating a target output for a first training input, wherein the target output comprises a historical value of the performance metric based on a historical threshold condition of the memory access operation performed by the prior memory device using a second historical set of values of the memory configuration parameters,… (Ummethala: ¶[0004]…Ummethala pairs each training input with a target output consisting of the historical measured performance values recorded for that same prior workpiece, which constitutes the recited historical performance-metric target output under BRI, “Generating the training data further includes generating a first target output for the first training input, wherein the first target output includes historical metrology measurements associated with the prior substrate previously processed at the manufacturing system”));
providing the training data to the trainable model on (i) a set of training inputs comprising the first training input and (ii) a set of target outputs comprising the target output (Ummethala: ¶[0004]…Ummethala provides the assembled training data to the model as a set of training inputs that includes the first training input paired with a set of target outputs that includes the first target output, which constitutes the recited providing step under BRI, “The method further includes providing the data to train the machine learning model on (i) a set of training inputs including the first training input, and (ii) a set of target outputs including the first target output”)).
Zhang and Ummethala are analogous art because they are from the same field of endeavor, specifically the training of machine-learning models on historical measurement data gathered from previously manufactured articles in order to predict process or device parameters. They are further reasonably pertinent to the particular problem the inventor faced, namely how to assemble a labeled training corpus for such a model from devices already produced by the manufacturing line.
Before the effective filing date of the claimed invention, it would have been obvious to a PHOSITA to generate the training data for Zhang’s deep neural network engine in the manner Ummethala teaches, that is, by forming each training input from historical parameter data recorded for a prior device previously produced at the manufacturing system and pairing it with a target output consisting of that prior device’s historical measured performance. This is the use of a known technique to improve a comparable method in the same way under MPEP 2143.01(C), yielding the predictable result of a labeled corpus assembled from production history rather than from fresh measurement.
The suggestion/motivation for doing so would have been provided by Ummethala itself, which teaches that historical measurements already collected for every article the manufacturing system has processed are available to serve as the model’s target outputs, so that no additional measurement campaign is required to build the training set, “Historical metrology measurements may be generated for each substrate processed at the manufacturing system. The historical metrology measurements may be provided as target outputs for the machine learning model” (Ummethala: ¶[0019]).
Per claim 11, Zhang combined with Ummethala discloses claim 10. Zhang further teaches identifying a value of a sub-system performance metric for a memory sub-system, wherein the memory sub-system comprises the memory device, and wherein the sub-system performance metric comprises the performance metric for the memory device (Zhang: ¶[0056]…Zhang identifies the performance of the data storage device - the solid-state drive that contains the memory dies - in terms of its input/output operations per second and throughput, and states that this drive-level performance is governed by the read threshold setting applied to the constituent memory device, so the drive-level figure so identified constitutes the recited sub-system performance metric that comprises the memory device’s performance metric under BRI, “The performance (e.g., input/output operations per second and throughput) of a data storage device such as an SSD is heavily dependent on the read threshold setting (i.e., read voltage setting) applied when the first read operation is conducted. If the read threshold is not optimized, the performance may be degraded because such unoptimized read threshold voltages can cause read errors”)).
Per claim 12, Zhang combined with Ummethala discloses claim 11. Zhang further teaches programming the historical set of values of the memory configuration parameters to a data structure comprised by the memory sub-system (Zhang: ¶[0065]…Zhang writes the read threshold values associated with the labeled training data set into a memory of the storage device that contains the memory device, which constitutes the recited programming of the historical parameter values to a data structure of the memory sub-system under BRI, “In an implementation, the optimized read thresholds associated with the labeled training data set can be stored in a memory (e.g., SRAM) in the storage device”)).
Per claim 13, Zhang combined with Ummethala discloses claim 10. Zhang does not expressly disclose, but Ummethala does teach: wherein each training input of the set of training inputs maps to a corresponding target output of the set of target outputs (Ummethala: ¶[0045]…Ummethala forms an explicit input/output mapping in which each training input is associated with the target output generated for it before the pair is added to the training set, which constitutes the recited mapping under BRI, “The input/output mapping refers to the training input that includes or is based on data for the substrate, and the target output for the training input, where the target output identifies a metrology measurement value for the substrate, and where the training input is associated with (or mapped to) the target output”)). The rationale to combine Ummethala with Zhang is the same as the parent claim.
Per claim 14, Zhang combined with Ummethala discloses claim 10. Zhang further teaches wherein the trainable model comprises at least one of a regression model, a neural network, a decision tree model, or a rule engine (Zhang: ¶[0058]…Zhang’s trainable model is a deep neural network trained to predict the optimal read threshold, which constitutes the neural network alternative; the claim recites the model types in the alternative, so disclosure of one alternative satisfies the limitation under BRI, “In some embodiments of the disclosed technology, a deep neural network is used to predict the optimal read threshold from the operating conditions of the storage device”)).
Per claim 15, Zhang combined with Ummethala discloses claim 10. Zhang further teaches wherein the memory configuration parameters comprise at least one of: a folding threshold, a forgiveness threshold, a bit error rate (BER), a residual bit error rate (RBER), an error trigger rate, a read refresh rate, a quality of service (QoS) threshold, a block family error avoidance (BFEA) bin pointer, a deep-check threshold, a read level offset threshold, a read disturb handling (RDH) threshold or a temperature compensation value (Zhang: ¶[0056]…Zhang’s memory configuration parameters are read threshold voltages - the read voltage setting applied when a read operation is conducted, “The performance (e.g., input/output operations per second and throughput) of a data storage device such as an SSD is heavily dependent on the read threshold setting (i.e., read voltage setting) applied when the first read operation is conducted. If the read threshold is not optimized, the performance may be degraded because such unoptimized read threshold voltages can cause read errors”; ¶[0055]…and Zhang modifies those read thresholds away from their nominal position to compensate the threshold-voltage shift that operating conditions produce, which is precisely the offset function the recited “read level offset threshold” alternative performs, “Such read errors, however, can be minimized by modifying the read thresholds. In some embodiments of the disclosed technology, the read thresholds may be modified based on operating conditions that contribute to the read errors in flash memory-based data storage SSD devices”))
Per claim 16, Zhang discloses A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising (Zhang: ¶[0073]…Zhang states that the operations it describes may be embodied as modules of computer program instructions encoded on a tangible and non-transitory computer readable medium and executed by a data processing apparatus, which constitutes the recited medium under BRI, “Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus”)):
providing, as an input to a trainable model, a value of a performance metric based on a threshold condition of a memory access operation performed on a memory device using a set of values of memory configuration parameters (Zhang: ¶[0060]…Zhang’s already-trained deep neural network receives at its input nodes the operating conditions that contribute to read errors, while the underlying data is measured on the memory device using a set of read thresholds, which constitutes the recited input under BRI, “The already-trained deep neural network 740, through the plurality of input nodes 710, the first connection layer 720, and the first connection nodes 730, receives the operating conditions that contribute to the read errors, such as endurance, retention, read disturbance, die index, block index, word line index, age of the data storage drive, and/or temperature. The deep neural network 740 measures data in the memory devices under combinations of operating conditions using a set of read thresholds”; ¶[0045]…Zhang defines endurance, one of those received conditions, as the maximum number of program/erase operations the device is guaranteed to perform successfully, which is a threshold-conditioned performance metric, “The endurance of flash memories may be represented by the maximum amount of program/erase operations that the flash memories are guaranteed to be able to perform program/erase operations successfully”));
obtaining as an output from the trainable model, a set of predicted values of the memory configuration parameters (Zhang: ¶[0066]…Zhang obtains the optimal read thresholds from the output nodes of the trained network, which constitutes the recited predicted parameter values under BRI, “At step 1010, the current operating conditions are fed into the input nodes of the trained deep neural network. At step 1020, the optimal read thresholds are obtained from output nodes of the trained deep neural network”)); and
…updating the memory configuration parameters to reflect the set of predicted values of the memory configuration parameters (Zhang: ¶[0066]…Zhang modifies the in-use read thresholds to the predicted optimal values and thereafter reads using them, which constitutes the recited updating step under BRI, “At step 1030, read thresholds are modified to the optimal read thresholds. At step 1040, data can be read out from the memory cell using the optimal read thresholds”)).
Zhang does not expressly disclose, but Ummethala does teach:
responsive to determining that the set of predicted values of the memory configuration parameters satisfies a confidence criterion,… (Ummethala: ¶[0073]…Ummethala adopts a predicted value for the workpiece under test only where the level of confidence in it satisfies a threshold condition, which constitutes the recited confidence-criterion condition under BRI, “At block 650, processing logic uses confidence data to estimate a metrology value for the substrate being processed at the manufacturing system. In some embodiments, if the level of confidence for a metrology value satisfies a threshold condition, then a substrate is identified as being associated with the metrology value”)).
Zhang and Ummethala are analogous art because they are from the same field of endeavor, specifically the machine-learning-based prediction of operating parameter values for semiconductor devices from measured performance data. They address the same problem of arriving at parameter values that cause a manufactured device to satisfy its performance requirements without exhaustive manual measurement of every operating combination.
Before the effective filing date of the claimed invention, it would have been obvious to a PHOSITA to embody Zhang’s predict-and-write-back sequence as instructions on the non-transitory medium Zhang itself contemplates, and to gate the write-back on the model’s confidence in the predicted values as Ummethala teaches. This is the use of a known technique to improve a comparable article of manufacture in the same way under MPEP 2143.01(C), yielding the predictable result of a stored program that declines to commit unreliable predicted parameter values.
The suggestion/motivation for doing so would have been provided by Ummethala itself, which teaches that a predicted value is adopted only where the model’s confidence in that value clears a threshold, “At block 650, processing logic uses confidence data to estimate a metrology value for the substrate being processed at the manufacturing system. In some embodiments, if the level of confidence for a metrology value satisfies a threshold condition, then a substrate is identified as being associated with the metrology value” (Ummethala: ¶[0073]).
Per claim 19, Zhang combined with Ummethala discloses claim 16. Zhang further teaches wherein the trainable model comprises at least one of a regression model, a neural network, a decision tree model, or a rule engine (Zhang: ¶[0058]…Zhang’s trainable model is a deep neural network trained to predict the optimal read threshold, which constitutes the neural network alternative under BRI, “In some embodiments of the disclosed technology, a deep neural network is used to predict the optimal read threshold from the operating conditions of the storage device”)).
Per claim 20, Zhang combined with Ummethala discloses claim 16. Zhang further teaches wherein the memory configuration parameters comprise at least one of: a folding threshold, a forgiveness threshold, a bit error rate (BER), a residual bit error rate (RBER), an error trigger rate, a read refresh rate, a quality of service (QoS) threshold, a block family error avoidance (BFEA) bin pointer, a deep-check threshold, a read level offset threshold, a read disturb handling (RDH) threshold or a temperature compensation value (Zhang: ¶[0056]…Zhang’s memory configuration parameters are read threshold voltages - the read voltage setting applied when a read operation is conducted, “The performance (e.g., input/output operations per second and throughput) of a data storage device such as an SSD is heavily dependent on the read threshold setting (i.e., read voltage setting) applied when the first read operation is conducted. If the read threshold is not optimized, the performance may be degraded because such unoptimized read threshold voltages can cause read errors”; ¶[0055]…and Zhang modifies those read thresholds away from their nominal position to compensate the threshold-voltage shift that operating conditions produce, which is precisely the offset function the recited “read level offset threshold” alternative performs, “Such read errors, however, can be minimized by modifying the read thresholds. In some embodiments of the disclosed technology, the read thresholds may be modified based on operating conditions that contribute to the read errors in flash memory-based data storage SSD devices”)).
Claims 3 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Ummethala and further in view of US Pat. Pub. No. 2022/0366280 to Rowe et al. (hereinafter Rowe).
Per claim 3, Zhang combined with Ummethala discloses claim 2. Zhang combined with Ummethala does not expressly disclose, but Rowe does teach:
providing, to a user interface device, the output from the trainable model as one or more numerical indicators (Rowe: ¶[0044]…Rowe’s data application layer renders the model’s prediction on a graphical user interface of a user interface device, and Rowe’s prediction is a numeric quantity accompanied by a numeric confidence value, which constitutes the recited providing of the model output to a user interface device as numerical indicators under BRI, “In another embodiment, the data application layer 130 may display on a graphical user interface (GUI) of the user interface 136 the prediction and the confidence value”)); and
providing, to the user interface device, the value of the confidence indicator (Rowe: ¶[0044]…Rowe displays the confidence value alongside the prediction on the same graphical user interface, and further exposes a user-interaction element by which the operator acts on or declines the prediction in light of it, which constitutes the recited providing of the confidence indicator value to the user interface device under BRI, “In another embodiment, the data application layer 130 may display on a graphical user interface (GUI) of the user interface 136 the prediction and the confidence value”)).
Zhang, Ummethala and Rowe are analogous art. Rowe is reasonably pertinent to the particular problem with which the inventor was concerned, namely how to surface a machine-learning model’s predicted values and the confidence attaching to them to the human operator who must decide whether to act on them; Rowe addresses that problem directly and is not confined to any one application domain.
Before the effective filing date of the claimed invention, it would have been obvious to a PHOSITA to present the predicted parameter values obtained from Zhang’s network, and the level of confidence Ummethala attaches to them, on a graphical user interface in the manner Rowe teaches. This is the use of a known technique to improve a comparable system in the same way under MPEP 2143.01(C), yielding the predictable result that the operator sees both the predicted values and the confidence attaching to them before the values are committed.
The suggestion/motivation for doing so would have been provided by Rowe itself, which teaches that the operator is given both the prediction and its confidence value together precisely so that the operator can decide whether to act on the prediction, “In another embodiment, the data application layer 130 may display on a graphical user interface (GUI) of the user interface 136 the prediction and the confidence value” (Rowe: ¶[0044]); Rowe further teaches withholding action when the confidence does not clear the threshold, “In the alternative, if the confidence value for the prediction does not meet a threshold value, the controller 133 may refrain from altering operations of one or more applications 134 or system operations 135 based on the prediction” (Rowe: ¶[0044]).
Per claim 17, Zhang combined with Ummethala discloses claim 16. Zhang does not expressly disclose, but Ummethala does teach:
updating a value of a confidence indicator, wherein the value of the confidence indicator reflects whether the confidence criterion was satisfied (Ummethala: ¶[0072]…Ummethala extracts a bounded level of confidence from the model output on each prediction, “At block 640, processing logic extracts confidence data from the outputs obtained at block 630. In some embodiments, the confidence data includes a level of confidence that a profile of the substrate is associated with a metrology value. In one example, the level of confidence is a real number between 0 and 1 inclusive”; ¶[0073]…and evaluates that value against the threshold condition to decide whether the predicted value is adopted, which constitutes the recited confidence indicator under BRI, “At block 650, processing logic uses confidence data to estimate a metrology value for the substrate being processed at the manufacturing system. In some embodiments, if the level of confidence for a metrology value satisfies a threshold condition, then a substrate is identified as being associated with the metrology value”));
Zhang combined with Ummethala does not expressly disclose, but Rowe does teach:
providing, to a user interface device, the output from the trainable model as one or more numerical indicators (Rowe: ¶[0044]…Rowe renders the model’s prediction on a graphical user interface of a user interface device, which constitutes the recited providing of the model output to a user interface device under BRI, “In another embodiment, the data application layer 130 may display on a graphical user interface (GUI) of the user interface 136 the prediction and the confidence value”)); and
providing, to the user interface device, the value of the confidence indicator (Rowe: ¶[0044]…Rowe displays the confidence value on that same interface alongside the prediction, which constitutes the recited providing of the confidence indicator value under BRI, “In another embodiment, the data application layer 130 may display on a graphical user interface (GUI) of the user interface 136 the prediction and the confidence value”)).
The rationale to combine Rowe with Zhang and Ummethala is the same as provided for claim 3.
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Ummethala, as applied in the rejection of claim 1 above, and further in view of US Pat. Pub. No. 2020/0387835 to Sandepudi et al. (hereinafter Sandepudi).
Per claim 6, Zhang combined with Ummethala discloses claim 1. Zhang combined with Ummethala does not expressly disclose, but Sandepudi does teach: the trainable model is a decision tree model comprising a set of branches, wherein a first branch set of branches comprises the threshold condition,… (Sandepudi: ¶[0058]…Sandepudi’s trained classifier is built from decision trees whose branches are threshold tests on feature values - a branch is satisfied when a feature exceeds or falls below a stated value - which constitutes the recited branch comprising the threshold condition under BRI, “This process will result in a sub-set of the 20,000 transactions for which Possible Candidate Rule #1 is satisfied (when Feature X is greater than 3.7, Feature Y is greater or equal to 0.82329, and Feature Z is less than 5.0002)”)); …wherein a first leaf of the decision tree model comprises the output from the trainable model and a branch path, the branch path indicating a series of branches of the set of branches between the input to the trainable model and the output from the trainable model (Sandepudi: ¶[0064]…Sandepudi traverses a path from a starting node to an ending node within a candidate tree and records the series of feature-value branches along that traversal, which is precisely the recited branch path indicating the series of branches between input and output, “Evaluating a branch path can thus include traversing a path from a starting node to an ending node within the given candidate tree, and constructing a possible candidate rule for the traversed path based on a feature value for each respective node in the traversed path”; ¶[0046]…and Sandepudi’s trees examine attribute values along those branches to reach a final assessment (score) at the terminating node, which constitutes the recited first leaf comprising the model output, “As noted above, a trained classifier can include many different decision trees. These trees may examine different data attributes in different combinations and values to reach a final assessment (score)”).
Zhang combined with Ummethala further discloses the confidence criterion of the second recited branch, as set forth in the rejection of claim 1 above; applying that criterion as a branch test within Sandepudi’s tree is the mechanical consequence of the combination.
Zhang, Ummethala and Sandepudi are analogous art. Sandepudi is reasonably pertinent to the particular problem with which the inventor was concerned, namely the selection and internal structuring of a trainable model so that the conditions by which it reaches an output can be enumerated and inspected. Sandepudi expressly disclaims confinement to its own example domain.
Before the effective filing date of the claimed invention, it would have been obvious to a PHOSITA to implement the trainable model of Zhang as combined with Ummethala as the decision-tree structure Sandepudi teaches, with the traversed branch path recorded alongside the output. This is the simple substitution of one known machine-learning model architecture for another under MPEP 2143.01(B), yielding the predictable result of a model that produces the same predicted parameter values while exposing the sequence of threshold tests by which it reached them.
The suggestion/motivation for doing so would have been provided by Sandepudi itself, which teaches that traversing and recording a tree’s branch path is what converts an otherwise opaque model output into an inspectable rule, “Evaluating a branch path can thus include traversing a path from a starting node to an ending node within the given candidate tree, and constructing a possible candidate rule for the traversed path based on a feature value for each respective node in the traversed path” (Sandepudi: ¶[0064]), and which expressly states that its techniques are not confined to its example domain, “Note that while certain techniques discussed herein are described relative to the example of transaction fraud detection, but techniques are generalizable to other kinds of machine learning environments and classification problems” (Sandepudi: ¶[0021]).
Per claim 7, Zhang combined with Ummethala discloses claim 1. Zhang combined with Ummethala does not expressly disclose, but Sandepudi does teach: the trainable model is a rule engine evaluating a set of rules, wherein a first rule of the set of rules comprises the threshold condition,… (Sandepudi: ¶[0058]…Sandepudi’s classification platform evaluates a set of rules, each rule being a conjunction of threshold conditions on feature values that a data item either satisfies or does not, which constitutes the recited rule engine and its first rule under BRI, “This process will result in a sub-set of the 20,000 transactions for which Possible Candidate Rule #1 is satisfied (when Feature X is greater than 3.7, Feature Y is greater or equal to 0.82329, and Feature Z is less than 5.0002)”)); wherein a second rule of the set of rules comprises the confidence criterion, wherein a first action of a set of actions comprises determining the value of the performance metric (Sandepudi: ¶[0059]…Sandepudi evaluates each rule against a second criterion - the precision and recall the rule achieves against a held-out evaluation set, tested against stated default values - and the action the engine performs on applying a rule is to compute that performance measure, which constitutes the recited second rule comprising the confidence criterion and the recited first action determining the performance-metric value under BRI, “One or more particular evaluation criteria can be used when evaluating a candidate rule. Precision and recall can be used as evaluation criteria; in some embodiments, a default value of 0.5 for precision and 0.1 for recall can be used”))
The rationale to combine Sandepudi with Zhang and Ummethala is the same as claim 6.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Ummethala, as applied in the rejection of claim 16 above, and further in view of US Pat. Pub. No. 2023/0061920 to Bhardwaj.
Zhang combined with Ummethala discloses claim 16. Zhang combined with Ummethala does not expressly disclose, but Bhardwaj does teach: transmitting, to a processing device, an instruction to initiate a validation check on the memory device, wherein the validation check determines whether the memory access operation performed on the memory device using the set of predicted values of the memory configuration parameters satisfies the threshold condition (Bhardwaj: ¶[0015]…Bhardwaj’s controller instructs the memory device to execute a read verify - a read performed on the device using its in-force parameters, whose error count is returned and tested against a threshold criterion - which constitutes the recited instruction to initiate a validation check that determines whether the operation satisfies the threshold condition under BRI, the parameters in force being the predicted values Zhang wrote back at step 1030, “A read verify operation is performed by initiating a read of a portion of the copyback data on the memory device. In embodiments, the memory device can perform the read on the portion of the copyback data, determine a number of errors in the portion read, and notify the memory sub-system controller of the same. The memory sub-system controller can determine whether the number of errors satisfies a read verify threshold criterion”).
Zhang, Ummethala and Bhardwaj are analogous art because all three are from the same field of endeavor, specifically the machine-learning-driven adjustment and verification of memory device operating parameters. They address the same problem of confirming that a device operating under newly applied parameter values in fact meets its error-rate requirement.
Before the effective filing date of the claimed invention, it would have been obvious to a PHOSITA to add to Zhang as combined with Ummethala the step Bhardwaj teaches of instructing the memory device to run a read-back check whose error count is tested against a threshold criterion, applied after Zhang has modified the read thresholds to the predicted optimal values. This is the use of a known technique to improve a comparable method in the same way under MPEP 2143.01(C), yielding the predictable result of confirming that the newly written parameter values in fact produce reads satisfying the error threshold.
The suggestion/motivation for doing so would have been provided by Bhardwaj itself, which teaches that reading back a portion of the data and testing the resulting error count against a threshold criterion is what prevents errors from propagating when data is committed under the parameters then in force, “To avoid error propagation, memory sub-systems can employ a read verify operation” (Bhardwaj: ¶[0015]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN CHEN whose telephone number is (571)272-4143. The examiner can normally be reached M-F 10-7.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN CHEN/Primary Examiner, Art Unit 2125
Prosecution Insights