DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Claim Objections
Claims 1-12 is/are objected to because of informalities. The examiner recommends the following changes.
Claim 1, line(s) 2, the acronym “DOI” should be spelled out to “defect of importance (DOI)” upon its first use.
Claim 9, “DOI” should be spelled out like claim 1.
Claims 1-8 and 10-12 depend either directly or indirectly from the objection of claims 1 and 9, therefore they are also objected.
Appropriate correction is required.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a model generating device,” “an image acquisition unit that acquires an observation image,” “information processing device that calculates,” “a storage unit that stores,” and “a calculation unit that executes first to fourth steps” in claim 9.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The corresponding structure for the image acquisition unit are the Fig. 2B optics and image sensor, or an electron beam inspection device (SPEC ¶¶ 13, 15). The information processing device corresponds to a processor and memory (SPEC ¶ 14) running Fig. 4’s algorithm (SPEC ¶¶ 19-20). The model generating device and calculation unit corresponds to a processor that executes a program loaded into memory (SPEC ¶ 28) to perform the algorithm of Figs. 5A and 5B described at SPEC ¶¶ 21-27.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 9, and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al., US 2019/0370955 A1 (hereinafter “Zhang”) in view of Hall et al., US 2022/0343178 A1 (hereinafter “Hall”).
Claim 1
Zhang discloses a model generating method for generating a deep learning model for classifying defect candidate images into DOI and Nuisance (Zhang ¶ 57: “The computer subsystem(s) are configured for performing active learning for training a defect classifier … separated as described further herein into actual defects and DOIs and nuisances”; ¶ 97: “The defect classifier is configured for classifying defects detected on the specimen using the images generated by the imaging subsystem.”; ¶ 83: “the computer subsystem(s) train one or multiple models (any model including a deep neural network)”.), the model generating method comprising:
a first step of preparing a plurality of deep learning models (Zhang ¶ 83: “the computer subsystem(s) train one or multiple models (any model including a deep neural network … These trainings may include multiple independent trainings.”; ¶ 88: “Each of the branches can be a different model”; ¶ 57: “the data points may include raw output generated for the specimen by the detector and/or some information determined from the raw output such as detected events.”);
a second step of labeling some of the defect candidate images to create teaching images(Zhang ¶ 91: “The active learning also includes acquiring labels for the selected one or more data points … from a user who manually assigns the labels”; ¶ 83: “Starting with substantially limited labeled data, e.g., limited data points 300, the computer subsystem(s) train one or multiple models.”);
a third step of (Zhang ¶ 83: “Starting with substantially limited labeled data, e.g., limited data points 300, the computer subsystem(s) train one or multiple models … These trainings may include multiple independent trainings.”; ¶ 88: “Each of the branches can be a different model.” Each model is trained on the labeled data points, which updates its parameters with the teaching images; Zhang does not score the models or keep one.); and
a fourth step of classifying the defect candidate images that have not been labeled (Zhang ¶ 61: “the acquisition function is configured to select the one or more of the data points that have the highest uncertainty of being any known defect type … so that those data points can be sent for verification (labeling) as described further herein”; ¶ 85: “the computer subsystem(s) may generate sub-pool 312 of unlabeled data from pool 310 of data … The computer subsystem(s) may then apply the model(s) on the sub-pool of data.”; ¶ 86: “The CLIRs are generally quantified by the uncertainty definition in information theory such as entropy”; ¶ 98: “The NEF may simply be configured to separate nuisance events from any other type(s) of events thereby producing a set of defects.” The uncertainty is an entropy over the predicted class probabilities, and for the nuisance event filter the classes are nuisance events and the other events, so the selection rests on the predicted likelihood of the DOI class. Under the broadest reasonable interpretation the DOI-likeness is the model’s predicted probability of the DOI class, SPEC ¶ 24: “the model outputs the probability that an input image will be DOI.”),
wherein the second step and the third step are executed again after executing the fourth step (Zhang¶ 96: “the active learning includes repeating at least once a sequence of steps that includes the applying, acquiring, and generating steps … the new labeled data is appended to the initial data set and the loop may be continued as desired”; ¶ 83: “FIG. 3 illustrates one embodiment of steps that may be performed for Bayesian Deep Leaming (BDL) and Active Discovery … This figure presents the workflow for a single BDL acquisition iteration". Each acquisition iteration trains the models again on the appended labels, so the labeling and training steps are executed again after each selection.).
Zhang does not teach the stricken through limitations; however, Hall teaches a plurality of deep learning models “with different hyper parameters in advance” (Hall ¶ 166: “A variety of deep learning models are available each with different architectures … Deep Learning models may be assessed based on … which models provided the best validation accuracy … Training involves trying different combinations of model parameters and hyper-parameters.” The candidate networks are built on different architectures and trained with different combinations of hyper-parameters, and they exist before training starts, so they
are prepared in advance.). Hall also teaches dividing the labeled images into teaching images “and evaluation images” (Hall ¶ 152: “each randomization of the total useable dataset is split into typically 3 separate datasets”; ¶ 153: “The first set is the training dataset … These images are used by deep learning models and computer vision models … This dataset is used to validate or test the accuracy of the model created using the training dataset.” The training dataset is the teaching images and the validation dataset is the evaluation images, both drawn from the labeled images.). Hall further teaches “evaluating classification accuracy using the evaluation images after” updating the parameters of each model (Hall ¶ 177: “After each epoch, the model is run on the validation set, without any training taking place, to provide a measure of the progress in how accurate the model is”; ¶ 153: “This dataset is used to validate or test the accuracy of the model created using the training dataset.”). Finally, Hall teaches “and selecting a best model with the highest classification accuracy from among the plurality of deep learning models,” the best model then being the one used (Hall ¶ 166: “which models provided the best validation accuracy”;¶ 107: “The best performing model is then selected 146, using a selection criteria … Total Accuracy … The performance of the models on a set of validation images are measured with respect to the metric, and then the best performing models selected accordingly … for creating an Ensemble model if desired”; ¶ 178: “The validation set guides the choice of the overall model parameters, or hyperparameters.” The model that performs best on the validation images by total accuracy is the best model with the highest classification accuracy; ensembling is optional. Hall¶ 166: “These methods are suitable for image analysis.”) Hall is reasonably pertinent to this problem the inventors faced (SPEC ¶ 8: “it is necessary to design an optimal deep learning model for each image type to be classified in order to enable classification with high accuracy.”), choosing the deep learning model configuration that classifies a given image type most accurately.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to prepare the deep models of Zhang in advance as candidate networks of different architectures and hyper-parameters, hold out part of the labeled defect data as a validation set, keep the candidate with the best validation accuracy as in Hall, and run the classification of the unlabeled defect data and the uncertainty-based selection of Zhang with that kept model, because selecting the configuration with the best validation accuracy on held-out labeled images yields a final model whose uncertainty estimates and DOI or nuisance decisions come from the configuration measured most accurate on data it was not trained on. (Hall ¶ 153: “Even though these images are independent of the training dataset used to create the model”; ¶ 107: “The performance of the models on a set of validation images are measured with respect to the metric, and then the best performing models selected accordingly.” Zhang¶ 83: “the computer subsystem(s) train one or multiple models (any model including a deep neural network”; ¶ 88: “Each of the branches can be a different model … each of the branches can be the same model”; ¶ 86: “The inferences may be performed with stochastic dropout to estimate CLIRs”; ¶ 90; “the collective model uncertainty across multiple models or multiple instances.” Zhang leaves the configuration of each model open, and its uncertainty can come from multiple dropout instances of one model. Zhang ¶ 83: "This figure presents the workflow for a single BDL acquisition iteration"; ¶ 96: “the active learning includes repeating at least once a sequence of steps that includes the applying, acquiring, and generating steps.” Because Zhang trains its models in every acquisition iteration, the Hall training, validation, and selection run again in each iteration, so the third step is executed again after the fourth step.) This is a teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the reference (MPEP 2143(G)). One of ordinary skill in the art would have had a reasonable expectation of success in making this modification. Zhang already allows training one model instead of several and obtaining uncertainty from multiple stochastic-dropout inferences of that one model rather than from an ensemble. Hall trains the candidate configurations on a training split, measures each on a validation split, and keeps the best, and the kept model then plays the role of Zhang's single model, including dropout-based uncertainty. KSR lnt'I Co. v. Teleflex Inc., 550 U.S. 398 (2007).
Claim 9
Zhang discloses a defect inspection system comprising:
a defect inspection device (Zhang ¶ 27: “The energy source is configured to generate energy that is directed to a specimen.”; ¶ 55: “any output that is above the one or more
thresholds may be identified by the computer subsystem(s) as defects or potential defects”; ¶ 28: “In one embodiment, the specimen includes a wafer.”); and
a model generating device that is connected to the defect inspection device and configured to generate a deep learning model for classifying defect candidate images acquired by the defect inspection device into DOI and Nuisance (Zhang ¶ 44: “computer subsystem 36 may be coupled to computer subsystem(s) 102 as shown by the dashed line in FIG. 1”; ¶ 57: “The computer subsystem(s) are configured for performing active learning for training a defect classifier … separated as described further herein into actual defects and DOIs and nuisances”; ¶ 83: “the computer subsystem(s) train one or multiple models (any model including a deep neural network)”. Computer subsystem(s) 102, coupled to the computer subsystem of the inspection tool, train the defect classifier.), wherein the defect inspection device includes an image acquisition unit that acquires an observation image (Zhang ¶ 27: “The energy source is configured to generate energy that is directed to a specimen.”; ¶ 29: “The imaging subsystem generates optical images by directing light to or scanning light over
the specimen and detecting light from the specimen … light from light source 16 is directed through optical element 18 and then lens 20”; ¶ 35: “the imaging subsystem may include stage 22 on which specimen 14 is disposed during imaging”; ¶ 36: “one formed by collector 24, element 26, and detector 28”; ¶ 27: “the specimen and to generate images responsive to the detected energy”), and
an information processing device that calculates a feature value of the observation image to extract defect candidate images and specifies, as defects, defect candidate images classified as DOT by using the deep learning model from among the extracted defect candidate images (Zhang ¶ 55: “any output that is above the one or more thresholds may be identified by the computer subsystem(s) as defects or potential defects”; ¶ 57: “one or more characteristics of the output (e.g., intensity values) corresponding to the detected events”; ¶ 98: “The NEF may simply be configured to separate nuisance events from any other type(s) of events thereby producing a set of defects … the output of the NEF filter may be
used as the inspection results”; ¶ 43: “any device having one or more processors, which executes instructions from a memory medium.” The intensity of the detector output compared to thresholds is the feature value, and the events the trained filter keeps are specified as defects.),
the model generating device includes a storage unit that stores (Zhang¶ 43: “any device having one or more processors, which executes instructions from a memory medium”; ¶ 83: “the computer subsystem(s) train one or multiple models (any model including a deep neural network)”; ¶ 88: “Each of the branches can be a different model.”),
the first step includes reading defect candidate images to be used for training the deep learning model (Zhang ¶ 57: “the data points may include raw output generated for the specimen by the detector and/or some information determined from the raw output such as detected events.”),
the second step includes labeling some of the defect candidate images to create teaching images and evaluation images (Zhang ¶ 91: “The active learning also includes acquiring labels for the selected one or more data points … from a user who manually assigns the labels”; ¶ 83: “Starting with substantially limited labeled data, e.g., limited data points 300, the computer subsystem(s) train one or multiple models.”),
the third step includes (Zhang ¶ 83: “Starting with substantially limited labeled data, e.g., limited data points 300, the computer subsystem(s) train one or multiple models … These trainings may include multiple independent trainings.”; ¶ 88: “Each of the branches can be a different model.”),
the fourth step includes classifying the defect candidate images that have not been labeled (Zhang ¶ 85: “the computer subsystem(s) may generate sub-pool 312 of unlabeled data from pool 310 of data … The computer subsystem(s) may then apply the model(s) on the sub-pool of data.”; ¶ 61: “the acquisition function is configured to select the one or more of the data points that have the highest uncertainty of being any known defect type … so that those data points can be sent for verification (labeling) as described further herein”; ¶ 86: “The CLIRs are generally quantified by the uncertainty definition in information theory such as entropy”; ¶ 98: “The NEF may simply be configured to separate nuisance events from any other type(s) of events thereby producing a set of defects.”), and the second step and the third step are executed again after executing the fourth step (Zhang ¶ 96: “the active learning includes repeating at least once a sequence of steps that includes the applying, acquiring, and generating steps … the new
labeled data is appended to the initial data set and the loop may be continued as desired”; ¶ 83: “This figure presents the workflow for a single BDL acquisition iteration.”).
Zhang does not teach the stricken through limitations; however, Hall teaches a plurality of deep learning models “with different hyper parameters” prepared “in advance” (Hall ¶ 166: “A variety of deep learning models are available each with different architectures … Deep Learning models may be assessed based on … which models provided the best validation accuracy … Training involves trying different combinations of model parameters and hyper-parameters.” The candidate networks are built on different architectures and trained with different combinations of hyper-parameters, and they exist before training starts, so they are prepared in advance.). Hall also teaches dividing the labeled images into teaching images “and evaluation images” (Hall ¶ 152: “each randomization of the total useable dataset is split into typically 3 separate datasets”; ¶ 153: “The first set is the training dataset … These images are used by deep learning models and computer vision models … This dataset is used to validate or test the accuracy of the model created using the training dataset.” The training dataset is the teaching images and the validation dataset is the evaluation images, both drawn from the labeled images.). Hall further teaches “evaluating classification accuracy using the evaluation images after” updating the parameters of each model (Hall ¶ 177: “After each epoch, the model is run on the validation set, without any training taking place, to provide a measure of the progress in how accurate the model is”; ¶ 153: “This dataset is used to validate or test the accuracy of the model created using the training dataset.”). Finally, Hall teaches “and selecting a best model with the highest classification accuracy from among the plurality of deep learning models,” the best model then being the one used (Hall ¶ 166: “which models provided the best validation accuracy”;¶ 107: “The best performing model is then selected 146, using a selection criteria … Total Accuracy … The performance of the models on a set of validation images are measured with respect to the metric, and then the best performing models selected accordingly … for creating an Ensemble model if desired”; ¶ 178: “The validation set guides the choice of the overall model parameters, or hyperparameters.” The model that performs best on the validation images by total accuracy is the best model with the highest classification accuracy; ensembling is optional. Hall ¶ 166: “These methods are suitable for image analysis.”) Hall is reasonably pertinent to this problem the inventors faced (SPEC ¶ 8: “it is necessary to design an optimal deep learning model for each image type to be classified in order to enable classification with high accuracy.”), choosing the deep learning model configuration that classifies a given image type most accurately.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to prepare the deep models of Zhang in advance as candidate networks of different architectures and hyper-parameters, hold out part of the labeled defect data as a validation set, keep the candidate with the best validation accuracy as in Hall, and run the classification of the unlabeled defect data and the uncertainty-based selection of Zhang with that kept model, because selecting the configuration with the best validation accuracy on held-out labeled images yields a final model whose uncertainty estimates and DOI or nuisance decisions come from the configuration measured most accurate on data it was not trained on. (Hall ¶ 153: “Even though these images are independent of the training dataset used to create the model”; ¶ 107: "The performance of the models on a set of validation images are measured with respect to the metric, and then the best performing models selected accordingly.” Zhang¶ 83: “the computer subsystem(s) train one or multiple models (any model including a deep neural network”; ¶ 88: “Each of the branches can be a different model … each of the branches can be the same model”; ¶ 86: “The inferences may be performed with stochastic dropout to estimate CLIRs”; ¶ 90; “the collective model uncertainty across multiple models or multiple instances.” Zhang leaves the configuration of each model open, and its uncertainty can come from multiple dropout instances of one model. Zhang ¶ 83: "This figure presents the workflow for a single BDL acquisition iteration"; ¶ 96: “the active learning includes repeating at least once a sequence of steps that includes the applying, acquiring, and generating steps.” Because Zhang trains its models in every acquisition iteration, the Hall training, validation, and selection run again in each iteration, so the third step is executed again after the fourth step.) This is a teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the reference (MPEP 2143(G)). One of ordinary skill in the art would have had a reasonable expectation of success in making this modification. Zhang already allows training one model instead of several and obtaining uncertainty from multiple stochastic-dropout inferences of that one model rather than from an ensemble. Hall trains the candidate configurations on a training split, measures each on a validation split, and keeps the best, and the kept model then plays the role of Zhang's single model, including dropout-based uncertainty. KSR lnt'I Co. v. Teleflex Inc., 550 U.S. 398 (2007).
Claim 12.
Zhang and Hall disclose the defect inspection system according to claim 9, wherein the image acquisition unit acquires the observation image based on a signal obtained by irradiating an inspection target sample with an electromagnetic wave or a charged particle beam (Zhang ¶ 29: “The imaging subsystem generates optical images by directing light to or scanning light over the specimen and detecting light from the specimen.”; ¶ 45: “the imaging subsystem is configured as an electron beam inspection subsystem”; ¶ 27: “the specimen and to generate images responsive to the detected energy.”).
Claim(s) 2 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Hall as applied to claims 1 and 9 above, and further in view of Hibi et al., JP 2020-154602 A (hereinafter “Hibi”). Citations to Hibi are to the English machine translation provided with this action.
Claims 2 and 10.
Zhang and Hall disclose the model generating method according to claim 1, wherein, [claim 2] / the defect inspection system according to claim 9, wherein [claim 10] when the best model selected in the third step (Zhang ¶ 96: “the active learning includes repeating at least once a sequence of steps that includes the applying, acquiring, and generating steps … the new labeled data is appended to the initial data set and the loop may be continued as desired.” Zhang repeats the selection and labeling but stops on an iteration count, a label count, or user feedback, not on accuracy.)
when the best model selected in the third stepI the best model is determined as a deep learning model to be used in the defect inspection device [claim 10] (Zhang ¶ 98: “The NEF may simply be configured to separate nuisance events from any other type(s) of events thereby producing a set of defects … the output of the NEF filter may be used as the inspection results.” The trained filter is the model used for the inspection results.)
Zhang and Hall do not teach the stricken through limitations; however, Hibi teaches a model that “does not satisfy a desired classification accuracy” and one that “satisfies the desired classification accuracy” (Hibi ¶ 73: “the learning unit 2 evaluates the identification accuracy of the trained model 17b whose identification ability has been improved in step S9 by using an evaluation image {marked image different from the image for learning)”; ¶ 74: “if a negative result is obtained in step S10, this means that the desired discrimination accuracy has not yet been obtained in the trained model 17b … the learning unit 2 moves to the next step S11 and repeats the above-described processing until a positive result is obtained in steps S4 and S10"; ¶ 66: "In step S11, the inference unit 4 selects the next unsupervised image from the other unsupervised images stored in the storage unit 3, proceeds to step S2 again.” A model short of the desired accuracy on the evaluation image returns the process to inferring unlabeled images and selecting the next ones to label. Hibi ¶ 75: “when an affirmative result is obtained in step S10, this means that the desired
identification accuracy is obtained in the trained model 17b … the learning unit 2 ends the above-mentioned active learning processing procedure.” A model that reaches it ends the loop and is the model used.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to stop the selection and labeling of Zhang once the kept model meets a desired accuracy on the evaluation images and continue it while the model falls short, as in Hibi, because Hibi evaluates the retrained model on a separately labeled evaluation image after each round and ends active learning once the desired accuracy is reached , so labeling continues only while the model falls short of the target accuracy and stops once it meets it, spending no labeling effort past the point of need. (Hibi ¶ 7: “capable of reducing the work load of the user and improving the identification accuracy of the trained model.”) This is a teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the reference (MPEP 2143(G)). One of ordinary skill in the art would have had a reasonable expectation of success in making this modification. The loop of Zhang already has a stopping condition, and the condition of Hibi replaces it with a check of accuracy on held-out evaluation images. Scoring a trained classifier on labeled evaluation images uses the same inference the combination already performs on the held-out images. KSR lnt'I Co. v. Teleflex Inc., 550 U.S. 398 (2007).
Claim(s) 5-6, 8, and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Hall as applied to claims 1 and 9 above, and further in view of Urano et al., US 2012/0229618 A1 (hereinafter “Urano”).
Claim 5
Zhang and Hall disclose the model generating method according to claim 1, wherein in the second step after the fourth step is executed, (Zhang ¶ 61: “the acquisition function is configured to select the one or more of the data points that have the highest uncertainty of being any known defect type”; ¶ 91: “from a user who manually assigns the labels”; Hall ¶ 166: "which models provided the best validation accuracy”; ¶ 97: “using the test dataset to select the best model.”).
Zhang and Hall do not teach the strikethrough limitations; however, Urano in the same field of endeavor teaches “a window for allowing a user to select whether some of the defect candidate images are DOI or Nuisance is displayed on a display device” (Urano ¶ 21: FIG. 6 shows an “exemplary defect candidate instructional GUI”; ¶ 84: “The defect matrix creation unit 504 displays the defect matrix in a defect matrix window (defect candidate window) 601 of the result display unit 407 … in an instruction menu 604 in the defect candidate instruction window 603”; ¶ 60: “the user manipulates an instruction input unit 408 to instruct that the defect candidate 307 is DOI or Nuisance.”). Urano also teaches “the defect candidate images displayed in the window are displayed in order of DOI-likenesses” (Urano ¶ 87: "The defect matrix shows arrays of defect candidate images each of which plots the group on the abscissa and the defect determination evaluation value on the ordinate."; ¶ 84: “The displayed defect matrix shows the defect candidates by plotting the group on the abscissa and the defect determination evaluation value on the ordinate.”; ¶ 74: “for the evaluation of the importance DOI of the defect candidate.” The candidate images are displayed along the ordinate in order of the defect determination evaluation value, which evaluates the importance DOI of each candidate; in the combination that value is the DOI likeness the best model assigns.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to present the defect candidate images that the kept model of Zhang and Hall selects for labeling, which as the candidates of highest uncertainty lie nearest the DOI and Nuisance boundary, in the defect matrix window of Urano, arranged by that model’s DOI-likeness with the DOI and Nuisance boundary drawn through them, and to take the DOI or Nuisance label from the instruction menu of Urano, because the selected defect data of Zhang are provided to a user for labeling without a defined presentation, and Urano arranges defect candidate images by DOI evaluation value with a boundary and a menu for marking each candidate, so the operator labels the flagged images in a view ordered by the model's DOI score, which reduces the number of candidates compared and lowers labeling burden and instruction time. (Urano ¶ 85: “There is also known a method of sampling only the defect candidates around the classification boundary 406. The sampling can reduce
the number of defect candidates which the user compares to give the instruction. This leads to the decrease in the burden on the user and the instruction time and to the improvement in the instruction correctness.”; ¶ 12: “it is not practicable for the user to check the defect candidates one by one and to give an instruction accordingly.”) This is a teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the reference (MPEP 2143(G)). KSR lnt'I Co. v. Teleflex Inc., 550 U.S. 398 (2007).
Claim 6
Zhang , Hall, and Urano disclose the model generating method according to claim 5, wherein the window is provided with a button for selecting whether the defect candidate images are DOT or Nuisance corresponding to the defect candidate images displayed in the window (Urano ¶ 84: “The defect matrix creation unit 504 displays the defect matrix in a defect matrix window (defect candidate window) 601 of the result display unit 407 … in an instruction menu 604 in the defect candidate instruction window 603”; ¶ 21: “exemplary defect candidate instructional GUI.” The user selects a displayed candidate in the window and then its DOI or Nuisance entry in the instruction menu, so the selection corresponds to the displayed image.).
Claim 8
The model generating method according to claim 1, wherein in the second step after the fourth step is executed, a window for allowing a user to specify whether some of the defect candidate images are DOT or Nuisance is displayed on a display device (Urano ¶ 21: “exemplary defect candidate instructional GUI”; ¶ 84: “The defect matrix creation unit 504 displays the defect matrix in a defect matrix window (defect candidate window) 601 of the result display unit 407 … in an instruction menu 604 in the defect candidate instruction window 603”; ¶ 60: “the user manipulates an instruction input unit 408 to instruct that the defect candidate 307 is DOI or Nuisance.” The defect matrix window and the instruction window are panes of one defect candidate instructional GUI), and the defect candidate images displayed in the window are displayed to be divided into a group evaluated to have a high DOT-likeness by the best model (Zhang ¶¶ 61, 91; Hall ¶ 166: “which models provided the best validation accuracy”; ¶ 97: “using the test dataset to select the best model.”) and a group evaluated to have a low DOT-likeness (Urano ¶ 87: “The defect matrix shows arrays of defect candidate images each of which plots the group on the abscissa and the defect determination evaluation value on the ordinate.”; ¶ 84: “The displayed defect matrix shows the defect candidates by plotting the group on the abscissa and the defect determination evaluation value on the ordinate. Furthermore, the defect matrix shows the classification boundary 406 between”; ¶ 74: “for the evaluation of the importance DOI of the defect candidate.” The classification boundary drawn across candidates plotted by evaluation value separates a high-value DOI group from a low-value Nuisance group.).
Claim 11
The defect inspection system according to claim 9, wherein the defect candidate images include an image of a background pattern (Urano ¶ 57: “clipped images (patch images) obtained from a detection image 307-2 and a reference image 307-3 … are collectively defined as a defect candidate 307”; ¶ 65: “similarity of the reference images (background image)”;
¶ 2: “a pattern defect or a defect such as foreign substance is detected based on a reference image and an inspection image.” The clipped reference image, which Urano calls the background image, is part of each defect candidate.) and an image of a defect candidate (Zhang ¶ 97: “The defect classifier is configured for classifying defects detected on the specimen using the images generated by the imaging subsystem.”; ¶ 57: “the data points may include raw output generated for the specimen by the detector and/or some information determined from the raw output such as detected events.”).
Allowable Subject Matter
Claims 3-4 and 7 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record but not relied, yet considered pertinent to the applicant’s disclosure, is listed on the PTO-892 form.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
