DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-4, 7-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Okuyama et al (US20220139788) in view of Steiman et al (US20220036538).
Regarding claim 1, Okuyama teaches a method of operating a transmission electron microscope (TEM) image processing apparatus, the method comprising:
acquiring a TEM image from a TEM facility;
(Okuyama, Fig. 1; “The evaluation apparatus 112 captures a cross section of the semiconductor ... acquires a cross-sectional image 208 as a processed result. The evaluation apparatus 112 includes a processing dimension measurement device using an SEM, a Transmission Electron Microscope (TEM), and an optical monitor”, [0062]; the system acquiring a cross-sectional image from a TEM facility)
Okuyama does not expressly disclose but Steiman teaches:
performing weak labeling of the TEM image to produce a partially labeled TEM image;
(Steiman, Fig. 6, “Partial label data associated with the training image can be provided by the user”, [0075], “The user annotation on the selected pixels in each segment can be used as the partial label data for training a machine learning model”, [0076]; “a group of pixels 606 within a structural element is marked by the user with a specific gray level value indicating that they belong to one segment representing the structural elements, and another group of pixels 608 in the background is marked by the user with another gray level value indicating that they belong to another segment representing the background area. The user annotation on the selected pixels in each segment can be used as the partial label data for training a machine learning model”, [0077]; partial/(weak) labeling by user)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Steiman into the system or method of Okuyama in order to allow users to mark only a few representative pixels instead of labeling the whole image, which saves time, reduces manual effort, and produces training data more efficiently for machine learning segmentation, making the system more practical and scalable. The combination of Okuyama and Steiman also teaches other enhanced capabilities.
The combination of Okuyama and Steiman further teaches:
generating ground truth for the partially labeled TEM image using a guide model;
(Steiman, Fig. 2A, “process the first training image using the trained ML model to obtain a first segmentation map informative of predicted labels associated with respective pixels in the first training image, each predicted label indicative of a segment that a respective pixel belongs to; and determine to include a first training sample comprising the first training image and the first segmentation map into the DNN training data upon a criterion being met, and to repeat the extracting of the second features, the training and the processing upon the criterion not being met”, [0006]; iterative ground truth generation using a trained model)
performing image segmentation using training data consisting of a pair of the TEM image and the ground truth;
(Steiman, “training data used for training a DNN (also referred to as DNN training data) can include a plurality of training samples, each including a respective training image and corresponding ground truth data associated therewith. Ground truth data can include label data indicative of application-specific information. By way of example, for the application of image segmentation, each training sample can include a training image of the semiconductor specimen and label data indicative of one or more segments in the training image”, [0066]; using an (image, ground truth) pair for segmentation model training)
measuring a device core structure in the TEM image according to a result of the segmentation; and
(Okuyama, Figs. 9 and 18; “obtains a bounding box coordinate that is a pattern detection result and a semantic segmentation image (step S106)”, [0073]; “Since the coordinates of the boundary lines of the regions are obtained from FIG. 18, the boundary line coordinates are cut out for each bounding box using the coordinate values of the bounding boxes in FIG. 17. Coordinates of the feature points A to G necessary for dimension measurement are obtained based on the cut-out boundary line coordinates of each bounding box, and the dimensions L1 to L5 are obtained”, [0108]; measurement using the result of segmentation)
displaying a measurement result according to the device core structure, and storing the measurement result in a database.
(Okuyama, “FIG. 19 shows an example in which the measured dimension values are displayed on the original input image, and the detected bounding box positions are also shown as rectangles”, [0109]; “The results are automatically stored in the database 205, and the average values are output to the input and output apparatus 206”, [0110])
Regarding claims 2 and 13, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the performing weak labeling comprises inputting one or several of a line, a connected line, a freedraw, a boundary, or a polygon to different regions of the TEM image, respectively.
(Steiman, Fig. 6, “a group of pixels 606 within a structural element is marked by the user with a specific gray level value indicating that they belong to one segment representing the structural elements”, [0076]; marking 606 => freedraw)
Regarding claim 3, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the guide model learns user input to generate segmentation ground truth.
(Steiman, “training a machine learning (ML) model using the first labels, values of the group of pixels selected in each segment associated with the first labels, and the feature values of each feature of the set of features corresponding to the group of pixels in each segment, wherein the ML model is trained for image segmentation; processing the first training image using the trained ML model to obtain a first segmentation map”, [0021]; user-added annotations guide the model to generate segmentation “ground truth.”)
Regarding claim 4, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 1,
wherein the generating ground truth comprises preparing a pre-learned guide model,
wherein the pre-learned guide model is one of a first guide model learned with a natural image, a second guide model learned with an image having a structure different from the device core structure, and a third guide model learned with the device core structure.
(Steiman, “Ground truth data can be obtained in various ways. By way of example, ground truth data can be produced by human annotation, synthetically produced (e.g. CAD-based images), generated by machine-learning annotation (e.g. labels based on feature extracting and analysis), or a combination of the above, etc.”, [0068]; use of multiple sources for model generation; implying the use of various sources (natural images, device images, synthetic data to train a model)
Regarding claim 7, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the TEM image and the ground truth are stored in the database as the training data.
(Steiman, Fig. 2A; “include a first training sample comprising the first training image and the first segmentation map into the DNN training data”, [0006]; storing image and segmentation results for use as training data)
Regarding claim 8, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the performing image segmentation comprises learning and inferring the TEM image and the ground truth using a segmentation model.
(Okuyama, “The central processing unit 201 passes the models and parameters from the database 205 to the model estimation unit 203, passes the input test image, and performs estimation by the model estimation unit 203 (step S105), and obtains a bounding box coordinate that is a pattern detection result and a semantic segmentation image (step S106)”, [0073]; learning/training and inference of a segmentation model with image/label pairs; Steiman, Figs. 2A and 2B; “the process as illustrated in FIGS. 2A and 2B can be repeated for one or more training images, so as to generate one or more segmentation maps, which, together with the training images, can be included in the DNN training data”, [0106]; “training data used for training a DNN (also referred to as DNN training data) can include a plurality of training samples, each including a respective training image and corresponding ground truth data associated therewith. Ground truth data can include label data indicative of application-specific information. By way of example, for the application of image segmentation, each training sample can include a training image of the semiconductor specimen and label data indicative of one or more segments in the training image”, [0066])
Regarding claim 9, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the measuring a device core structure comprises measuring the device core structure with respect to the TEM image of a semiconductor device in which a boundary between materials is distinguished.
(Okuyama, “A first image recognition model is an image recognition model that extracts a boundary line between a processing structure and a background over the entire cross-sectional image and/or a boundary line of an interface between different kinds of materials”, [0051])
Regarding claim 10, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 1, wherein the measured device core structure comprises measurement result values for width, height, and roughness.
(Okuyama, “the dimension measurement points include five parts of (1) L1: a mask upper surface width 305, (2) L2: a mask/substrate interface width 306, (3) L3: a substrate narrowest portion width 307, (4) L4: a mask height 308, and (5) L5: a trench depth 309”, [0096]; “FIG. 19 shows an example in which the measured dimension values are displayed on the original input image, and the detected bounding box positions are also shown as rectangles”, [0109]; width/height, and surface profile; roughness can be inferred from deviation/statistics of measured lines)
Regarding claim 11, the combination of Okuyama and Steiman teaches a method of operating a transmission electron microscope (TEM) image processing apparatus, the method comprising:
receiving a TEM image;
(Okuyama, see comments on claim 1)
adding user input to the TEM image using weak labeling to produce a partially labeled TEM image;
(Steiman, see comments on claim 1; Fig. 6, “The user annotation on the selected pixels in each segment can be used as the partial label data for training a machine learning model”, [0077]; partial/(weak) labeling by user)
The combination of Okuyama and Steiman further teaches:
generating ground truth for the partially labeled TEM image using a guide model;
determining whether the guide model satisfies a performance criterion; and
generating training data when the guide model satisfies the performance criterion,
(Steiman, Fig. 2A, “process the first training image using the trained ML model to obtain a first segmentation map informative of predicted labels associated with respective pixels in the first training image, each predicted label indicative of a segment that a respective pixel belongs to; and determine to include a first training sample comprising the first training image and the first segmentation map into the DNN training data upon a criterion being met, and to repeat the extracting of the second features, the training and the processing upon the criterion not being met”, [0006]; iterative ground truth generation using a trained model)
wherein the ground truth includes an image for distinguishing a boundary between materials.
(Okuyama, “A first image recognition model is an image recognition model that extracts a boundary line between a processing structure and a background over the entire cross-sectional image and/or a boundary line of an interface between different kinds of materials”, [0051])
Regarding claim 12, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 11, wherein the adding user input comprises inputting a point, a line, or a curve into one or more regions of the TEM image.
(Steiman, “the user can select a few pixels from either segment and mark them to be indicative of a respective segment”, [0076]; user input as points or more generally marking areas)
Regarding claim 14, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 11, wherein the guide model is updated using active learning that adds user input based on an inference result and an uncertainty map.
(Steiman, “upon receiving a negative user feedback on the first segmentation map, obtain additional first labels associated with an additional group of pixels in at least one of the segments... and repeat the extracting of the second features, the training and the processing based on the aggregated label data until receiving a positive user feedback”, [0007]; user input based on model result serves as active learning; anything is uncertain before receiving user’s positive feedback)
Regarding claim 15, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 14, further comprising storing the training data and the updated guide model in a database.
(Okuyama, “parameters of the learned models are stored in the database 205”, [0072]; storing both training data and models in a database; Steiman, “The classifier can be re-trained (216) using the aggregated labels”, [0103]; “whether to include the first training sample in the DNN training data, or to repeat the process... upon receiving a positive user feedback on the first segmentation map, the first training sample can be included into the DNN training data”, [0096]; stores training data and, by context, model metadata; the repeating processing implies updating the model)
Regarding claim 16, Okuyama teaches a method of operating a transmission electron microscope (TEM) image processing apparatus, the method comprising:
(Okuyama, Fig. 1; “The evaluation apparatus 112 captures a cross section of the semiconductor ... acquires a cross-sectional image 208 as a processed result. The evaluation apparatus 112 includes a processing dimension measurement device using an SEM, a Transmission Electron Microscope (TEM), and an optical monitor”, [0062]; the system acquiring a cross-sectional image from a TEM facility)
collecting training data consisting of a pair of a TEM image and ground truth corresponding to the TEM image;
(Steiman, “training data used for training a DNN (also referred to as DNN training data) can include a plurality of training samples, each including a respective training image and corresponding ground truth data associated therewith. Ground truth data can include label data indicative of application-specific information. By way of example, for the application of image segmentation, each training sample can include a training image of the semiconductor specimen and label data indicative of one or more segments in the training image”, [0066]; using an (image, ground truth) pair for segmentation model training; the training image may be a TEM image of Okuyama)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Steiman into the system or method of Okuyama in order to allow users to mark only a few representative pixels instead of labeling the whole image, which saves time, reduces manual effort, and produces training data more efficiently for machine learning segmentation, making the system more practical and scalable. The combination of Okuyama and Steiman also teaches other enhanced capabilities.
The combination of Okuyama and Steiman further teaches:
learning a segmentation model for distinguishing a boundary between materials in an image;
(Steiman, “the DNN is trained for a specific application of image segmentation on semiconductor specimen images (e.g., the FP images). The term “segmentation” used herein may refer to any process of partitioning an image into meaningful parts/segments (for example, background and foreground, noisy and non-noisy areas, structural elements, defect and non-defect, etc.)”, [0069])
determining whether the segmentation model satisfies a performance criterion;
(Steiman, Fig. 2A, “process the first training image using the trained ML model to obtain a first segmentation map informative of predicted labels associated with respective pixels in the first training image, each predicted label indicative of a segment that a respective pixel belongs to; and determine to include a first training sample comprising the first training image and the first segmentation map into the DNN training data upon a criterion being met, and to repeat the extracting of the second features, the training and the processing upon the criterion not being met”, [0006])
performing inference for the TEM image using the segmentation model, when the segmentation model satisfies the performance criterion;
(Steiman, Fig. 4, Runtime phase (420); “During runtime 420, PMC uses the trained DNN to process (422) one or more runtime images comprising one or more captured FP images to be segmented in order to provide (424) a segmentation map for each image. The one or more FP images can be obtained by the same or by different examination modalities. The obtained segmentation map can be informative of per-pixel or per-region segmentation labels indicative of different segments on the image”, [0111]; the runtime images may be the TEM images of Okuyama)
measuring a device core structure according to a result of the inference; and
outputting the device core structure.
(Okuyama, see comments on claim 1)
Regarding claim 17, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 16, wherein the training data is generated by weak labeling.
(Steiman, see comments on claim 1)
Regarding claim 18, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 16, wherein the learning a segmentation model is repeated until the segmentation model satisfies the performance criterion.
(Steiman, Fig. 2A, “process the first training image using the trained ML model to obtain a first segmentation map informative of predicted labels associated with respective pixels in the first training image, each predicted label indicative of a segment that a respective pixel belongs to; and determine to include a first training sample comprising the first training image and the first segmentation map into the DNN training data upon a criterion being met, and to repeat the extracting of the second features, the training and the processing upon the criterion not being met”, [0006]; iterative ground truth generation using a trained model)
Regarding claim 20, the combination of Okuyama and Steiman teaches its/their respective base claim(s).
The combination further teaches the method of claim 16,
wherein the device core structure comprises a measurement value for a width, a measurement value for a height, and a measurement value for a roughness, and
(Okuyama, see comments on claim 10)
wherein the method further comprises storing the measurement values for the width, the height, and the roughness in a database.
(Okuyama, “The results are automatically stored in the database 205, and the average values are output to the input and output apparatus 206”, [0110]; any data to be displayed usually are stored in memory first)
Allowable Subject Matter
Claim(s) 5-6 and 19 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening Claim(s).
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 5 and 19 recite(s) limitation(s) related to a model which outputs ground truth, overlays uncertainty map, updates with added user input, avoids overfitting; and user’s input of encoded client-side, decoded server-side, used to update segmentation model for TEM inference. There are no explicit teachings to the above limitation(s) found in the prior art cited in this office action and from the prior art search.
Claim(s) 6 depends on claims 5.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 1/11/2026