DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings were received on 02/20/2023. These drawings are acceptable.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on the following date(s): 11/17/2025, 08/15/2025 and 06/21/2023 have been considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: listed below, where the generic placeholder is in bold and the functional language italicize:
Claim 1:
An apparatus, comprising: an electron-beam column configured to scan an electron beam across a sample;
a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality;
and an electronic controller connected to receive streams of measurements from the plurality of detectors and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements, the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality; identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs; and generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image.
Claim 4:
wherein the autoencoder comprises: a neural network encoder configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in the latent space;
and a neural network decoder configured to generate reconstructed spectra based on mappings, with the neural network encoder, of training spectra to the latent space;
Claim 5
wherein the autoencoder comprises: a first neural network encoder configured to map the respective first input vector to a first probability density in a first private subspace of the latent space; and a second neural network encoder configured to map the respective second input vector to a second probability density in a second private subspace of the latent space; and wherein the first neural network encoder and the second neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space
claim 9
wherein the electronic controller is configured to generate the cluster-mapped image of the sample by coloring each pixel of the base image in accordance with a color code of the plurality of clusters
claim 10
wherein the electronic controller is configured to apply processing to the streams of measurements to obtain the respective first input vector and the respective second input vector, the processing including one or more operations selected from the group consisting of removal of outlier peaks, subtraction of an estimated background, scaling, normalization, averaging, fitting with a selected function, binning or re-binning, and Gaussian kernel filtering.
Claim 13:
an interface device configured to receive streams of measurements from a plurality of detectors of the scientific instrument, the plurality of detectors being configured to measure signals caused by interaction of an electron beam with a sample and including a first detector for a first spectroscopic modality of the scientific instrument, a second detector for a second spectroscopic modality of the scientific instrument, and a third detector for an imaging modality of the scientific instrument; and a processing device configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements, the respective first input vector corresponding to the first spectroscopic modality, the respective second input vector corresponding to the second spectroscopic modality; identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs; and generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image.
Claim 14:
wherein the autoencoder comprises: a neural network encoder configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in the latent space; and a neural network decoder configured to generate reconstructed spectra based on mappings, with the neural network encoder, of training spectra to the latent space; and wherein the neural network encoder and the neural network decoder have been trained using a loss function, the training spectra, and the reconstructed spectra, the loss function including a sum of a term representing reconstruction loss and a regularizer term.
Claim 15:
wherein the autoencoder comprises: a first neural network encoder configured to map the respective first input vector to a first probability density in a first private subspace of the latent space; and a second neural network encoder configured to map the respective second input vector to a second probability density in a second private subspace of the latent space; and wherein the first neural network encoder and the second neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space.
Claim 18:
further comprising a display device, wherein the processing device is configured to: generate the cluster-mapped image of the sample by coloring each pixel of the base image in accordance with a color code of the plurality of clusters; and cause the cluster-mapped image of the sample to be displayed by the display device.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, the term “cluster-mapped image” in the limitation “generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image” renders the claim definite because it is unclear what a clustered mapped image given that the claim notes that the mapping is completed using an autoencoder “for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder…” for mapping with respect to input vectors. Where are the clusters generated with respect to the autoencoder? And autoencoder generates an imaged by mapping elements to a latent space, as known in the art, but the term cluster-mapped image is not a term of art.
Examiner notes that any image generated using an autoencoder reads on the claimed limitation/term.
Regarding claims 13 and 19, the limitations are similar with claim 1 and thus rejected under the same rationale.
Regarding the dependent claims of claims 1, 13, and 19, the claims fail to resolve the noted deficiencies and the ones that recite the same term are deemed indefinite under the same rationale. The rejection noted above is incorporated.
Claim limitations 1, 4-5, 9-10, 13-15 and 18 are invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The noted generic place holder noted above, are noted to perform the claimed function where there is no recitation linking the generic place holder to the corresponding structure, material, or acts for performing the entire claimed function, noted above. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 10-13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (US 20180330511, hereinafter ‘Ha’) in view of Higuchi et al. (US 20220301288, hereinafter ‘Hig’).
Regarding independent claim 1, Ha teaches an apparatus, comprising: an electron-beam column configured to scan an electron beam across a sample; a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality; (in [0037] In some instances, the optical tool may be configured to direct light to the specimen [an apparatus, comprising: an electron-beam column configured to scan an electron beam across a sample] at more than one angle of incidence at the same time. For example, the illumination subsystem may include more than one illumination channel, one of the illumination channels may include light source 16, optical element 18, and lens 20 as shown in FIG. 1 and another of the illumination channels (not shown) may include similar elements, which may be configured differently or the same, or may include at least a light source and possibly one or more other components such as those described further herein. If such light is directed to the specimen at the same time as the other light, one or more characteristics (e.g., wavelength, polarization, etc.) of the light directed to the specimen at different angles of incidence may be different such that light resulting from illumination of the specimen at the different angles of incidence can he discriminated from each other at the detector(s) [plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample]…; And in [0052] As also shown in FIG. 1, the electron column includes electron beam source 126 configured to generate electrons that are focused to specimen 128 [plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample] by one or more elements 130. The electron beam source may include, for example, a cathode source or emitter tip, and one or more elements 130 may include, for example, a gun lens, an anode, a beam limiting aperture, a gate valve, a beam current selection aperture, an objective tens, and a scanning subsystem, all of which may include any such suitable elements known in the art… in [0067] As further noted above, the optical tool may be configured to generate output for the specimen with multiple modes or “different modalities [the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality].” In this manner, in some embodiments, the optical images include images generated by the optical tool with two or more different values of a parameter of the optical tool. In general, a “mode” or “modality” (as those terms are used interchangeably herein) of the optical tool can be defined by the values of parameters of the optical tool used for generating output and/or images for a specimen. Therefore, modes that are different may be different in the values for at least one of the optical parameters of the tool…)
and an electronic controller connected to receive streams of measurements from the plurality of detectors (in [0069] The optical and electron beam tools described herein may be configured as inspection tools [and an electronic controller connected to receive streams of measurements from the plurality of detectors]. In addition, or alternatively, the optical and electron beam tools described herein may be configured as defect review tools. Furthermore, the optical and electron beam tools described herein may be configured as metrology tools. In particular, the embodiments of the optical and electron beam tools described herein and shown in FIG. 1 may be modified in one or more parameters to provide different imaging capability [and an electronic controller connected to receive streams of measurements from the plurality of detectors] depending on the application for which they will be used. In one such example, the optical tool shown in FIG. 1… [0072] The one or more computer subsystems (e.g., computer subsystem(s) 36, 102, and 124 shown in FIG. 1) included in the system are configured for acquiring information for a specimen. The information for the specimen includes at least first and second images for the specimen. In the case of actual images, the computer subsystem may be configured for acquiring the actual images by using one or more of the tools [and an electronic controller connected to receive streams of measurements from the plurality of detectors] described herein for directing energy (e.g., light or electrons) to a specimen and detecting energy (e.g., light or electrons) from the specimen. Therefore, acquiring the actual images may include generating the images using a physical version of the specimen and some sort of imaging hardware. However, acquiring the actual images may include acquiring the actual images from a storage medium (including any of the storage media described herein) in Which the actual images have been stored by an actual imaging system (e.g., optical tool 10)...)
and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, (in [0074] The different modalities are different in at least one imaging parameter of at least one imaging system. In one embodiment, the first and second modalities generate the first and second images with different pixel sizes... In one such example, an image captured using an optical imaging system and an image captured using an electron beam imaging system are captured at different frequencies… [0091] Deep learning is part of a broader family of machine learning methods based on learning representations of data. An observation (e.g., an image) can be represented in many ways such as a vector of intensity values per pixel, or in a more abstract way as a set of edges, regions of particular shape, etc. Some representations are better than others at simplifying the learning task (e.g., face recognition or facial expression recognition)… [0110] As shown in FIG. 3, SEM image 300 (or a first image acquired for a specimen with a first modality) is input to learning based model 302, which transforms the SEM image to thereby render it into the common space of CAD image 306. In other words, learning based model 302 transforms SEM image 300 to rendered image 304 by mapping SEM image 300 from SEM image space to CAD image space [and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector]... Since the rendered image and the CAD image now look as if they were acquired from the same modality prior to alignment, alignment can be performed relatively easily as described further herein. [0111] In the embodiment shown in FIG. 3, the learning based model may be a regression model or any of the learning based models described herein. In one such example, the learning based model may be in the form of a deep convolution autoencoder (DCAE) [and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector]. The encoder portion of the learning based model may include, for example, five convolutional layers with kernel sizes of, for example, 5×5, a stride of 2, and no zero padding… [0115] The embodiment shown in FIG. 3 shows a run time mode of one of the alignment approaches described herein, FIG. 4 shows one possible method for training such an alignment approach. As shown in FIG. 4, the training may include inputting SEM image 400 into learning based model 402, which may be a regression model or another learning based model described herein. In this embodiment, the learning based model includes encoder 404 and decoder 408, ... Features 406 are input to decoder 408, which transforms the image into a different space. In this case, the decoder transforms the input SEM image from features 406 to image 410 in design space. In this manner, image 410 may be a CAD image.)
and the base image being obtained based on the streams of measurements, the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality; (in 0110] As shown in FIG. 3, SEM image 300 (or a first image acquired for a specimen with a first modality) is input to learning based model 302, which transforms the SEM image to thereby render it into the common space of CAD image 306. In other words, learning based model 302 transforms SEM image 300 to rendered image 304 by mapping SEM image 300 from SEM image space to CAD image space. In this manner, the common space in this embodiment is CAD image space. As such, in this embodiment, the second image is the CAD image generated for the specimen with a second modality. Rendered image 304 and CAD image 306 are then input to alignment step 308, which performs alignment or registration of the two images to thereby generate alignment results 310 [the base image being obtained based on the streams of measurements, the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality]… [0115] The embodiment shown in FIG. 3 shows a run time mode of one of the alignment approaches described herein, FIG. 4 shows one possible method for training such an alignment approach. As shown in FIG. 4, the training may include inputting SEM image 400 into learning based model 402, which may be a regression model or another learning based model described herein. In this embodiment, the learning based model includes encoder 404 and decoder 408 [with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector], Which may be configured as described further herein. In addition, although an auto-encoder is shown in this figure in the learning based model, any regression model such as CG AN or demise convolutional auto-encoder can be used in the embodiments described herein. Image 400 is input to encoder 404, which determines features 406 (i.e., learning or deep learning based features) of the image. Features 406 are input to decoder 408, which transforms the image into a different space. In this case, the decoder transforms the input SEM image from features 406 to image 410 in design space. In this manner, image 410 may be a CAD image.)
identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs; (As depicted in Fig. 4 and in [0115] The embodiment shown in FIG. 3 shows a run time mode of one of the alignment approaches described herein, FIG. 4 shows one possible method for training such an alignment approach. As shown in FIG. 4, the training may include inputting SEM image 400 into learning based model 402, which may be a regression model or another learning based model described herein. In this embodiment, the learning based model includes encoder 404 and decoder 408, Which may be configured as described further herein. In addition, although an auto-encoder is shown in this figure in the learning based model, any regression model such as CG AN or demise convolutional auto-encoder can be used in the embodiments described herein. Image 400 is input to encoder 404, which determines features 406 (i.e., learning or deep learning based features) [identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs] of the image. Features 406 are input to decoder 408, which transforms the image into a different space. In this case, the decoder transforms the input SEM image from features 406 [identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs] to image 410 in design space. In this manner, image 410 may be a CAD image.)
and generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image. (in [0115] The embodiment shown in FIG. 3 shows a run time mode of one of the alignment approaches described herein, FIG. 4 shows one possible method for training such an alignment approach. As shown in FIG. 4, the training may include inputting SEM image 400 into learning based model 402 [based on the base image and further based on latent-space clusters identified for different pixels of the base image], which may be a regression model or another learning based model described herein. In this embodiment, the learning based model includes encoder 404 and decoder 408 […based on the base image and further based on latent-space clusters identified for different pixels of the base image], Which may be configured as described further herein. In addition, although an auto-encoder is shown in this figure in the learning based model, any regression model such as CG AN or demise convolutional auto-encoder can be used in the embodiments described herein. Image 400 is input to encoder 404, which determines features 406 (i.e., learning or deep learning based features) of the image. Features 406 are input to decoder 408, which transforms the image into a different space. In this case, the decoder transforms the input SEM image from features 406 to image 410 in design space [generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image]. In this manner, image 410 may be a CAD image… [0117] The embodiments described above provide a number of differences and improvements compared to the currently used methods. For example, different from the currently used methods that are based on either heuristic renderings or physics-based rendering approaches, the embodiments described above uses a deep regression neural network or other learning based model described further herein trained with pairs of corresponding images from different modalities to transform image 1 to image domain of image 2 for registration, e.g., from SEM to CAD images, from SEM to broadband optical images, etc… And in [0145] Another embodiment relates to a computer-implemented method for aligning images for a specimen acquired with different modalities... The method also includes inputting the information for the specimen into a learning based model [… based on the base image and further based on latent-space clusters identified for different pixels of the base image]. The learning based model is included in one or more components executed by one or more computer systems… [0151] Further modifications and alternative embodiments of various aspects of the invention will be apparent to those skilled in the art in view of this description. For example, methods and systems for aligning images for a specimen acquired with different modalities are provided...)
Ha teaches processing image data using machine learning models, including autoencoder models. One of ordinary skill in the art would understand that the latent space of an autoencoder can model corresponding features using a probability density function/distribution.
Hig expressly teaches that the latent space of an autoencoder can model corresponding features using a probability density function/distribution, in [0080] A variational autoencoder 151 is one type of autoencoder. An autoencoder [for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, …] is a multilayer neural network that is created with machine learning such that input data and output data are identical to each other. The autoencoder compresses the input data into a vector having fewer dimensions than the input data, and restores the output data from the vector. Here, the variational autoencoder 151 is created such that a set of vectors follows a specific probability distribution [… a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements, the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality;…]. The variational autoencoder 151 includes an encoder 152 and a decoder 153. … [0082] A vector 155 is calculated between the encoder 152 and the decoder 153. The vector 155 is a representation of the features of the image 157 in low dimensions. For example, the vector 155 has 48 dimensions. The vector 155 may be called a latent variable, feature value, feature vector, or another. The vector 155 is mapped to a latent space 154. The latent space 154 is a vector space such as a 48-dimensional space. [0083] When a set of images of the same type (for example, a set of face photos or a set of handwritten characters) is input to the encoder 152, a set of vectors [… a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements… ] corresponding to the set of images has a specific probability distribution such as a normal distribution in the latent space 154 [… map, with an autoencoder, … identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs]. For example, the probability distribution in the latent space 154 is a multivariate normal distribution that has the vector 155 as a probability variable and that is specified by a specific mean vector and variance-covariance matrix. Here, a probability distribution other than the normal distribution may be assumed. The probability of occurrence of a specified vector in the set of vectors is approximated to a probability density calculated by a probability density function [… map, with an autoencoder, … identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs]. In general, a vector closer to the mean vector has a higher probability density, whereas a vector farther away from the mean vector has a lower probability density [identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs].
Hig and Ha are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data for a detection system based on feature vectors calculated by the neural network, as disclosed by Hig with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as disclosed by Ha.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Hig and Ha as noted above; Doing so allows for developing information processing and modeling techniques to reduce the possibility of generating image data that is clearly dissimilar to sample image data included in the training data and to reduce the possibility that an image converges to a local solution and the inference of the sample image data fails, (Hig, 0048).
Regarding claim 10, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein the electronic controller is configured to apply processing to the streams of measurements to obtain the respective first input vector and the respective second input vector, the processing including one or more operations selected from the group consisting of removal of outlier peaks, subtraction of an estimated background, scaling, normalization, averaging, fitting with a selected function, binning or re-binning, and Gaussian kernel filtering. (in [0112] To avoid overfitting and reduce redundancy in the extracted features, sparsity in the feature maps may be enforced by using a drop out layer at the end of the encoder and also including a L1 regularization on the codes in the L2 cost function [wherein the electronic controller is configured to apply processing to the streams of measurements to obtain the respective first input vector and the respective second input vector, the processing including one or more operations selected from the group consisting of …, fitting with a selected function]. Again, these specific learning based model configurations are not meant to be limiting to the learning based models that are appropriate for use in the embodiments described herein. The learning based model may vary in type and parameter values from those described above and still be used in the embodiments described herein.)
Regarding claim 11, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein the first detector is positioned downstream from the sample with respect to a propagation direction of the electron beam; and wherein the second detector is positioned upstream from the sample with respect to the propagation direction of the electron beam. (in [0037] In some instances, the optical tool may be configured to direct light to the specimen at more than one angle of incidence at the same time [wherein the first detector is positioned downstream from the sample with respect to a propagation direction of the electron beam; and wherein the second detector is positioned upstream from the sample with respect to the propagation direction of the electron beam]. For example, the illumination subsystem may include more than one illumination channel, one of the illumination channels may include light source 16, optical element 18, and lens 20 as shown in FIG. 1 and another of the illumination channels (not shown) may include similar elements, which may be configured differently or the same, or may include at least a light source and possibly one or more other components such as those described further herein. If such light is directed to the specimen at the same time as the other light, one or more characteristics (e.g., wavelength, polarization, etc.) of the light directed to the specimen at different angles of incidence may be different such that light resulting from illumination of the specimen at the different angles of incidence [wherein the first detector is positioned downstream from the sample with respect to a propagation direction of the electron beam; and wherein the second detector is positioned upstream from the sample with respect to the propagation direction of the electron beam] can he discriminated from each other at the detector(s)… [0044] Although FIG. 1 shows an embodiment of the optical tool that includes two detection channels, the optical tool may include a different number of detection channels (e.g., only one detection channel or two or more detection channels). In one such instance, the detection channel formed by collector 30, element 32, and detector 34 may form one side channel as described above, and the optical tool may include an additional detection channel (not shown) formed as another side channel that is positioned on the opposite side of the plane of incidence [wherein the first detector is positioned downstream from the sample with respect to a propagation direction of the electron beam; and wherein the second detector is positioned upstream from the sample with respect to the propagation direction of the electron beam]. Therefore, the optical tool may include the detection channel that includes collector 24, element 26, and detector 28 and that is centered in the plane of incidence and configured to collect and detect light at scattering angle(s) that are at or close to normal to the specimen surface [wherein the first detector is positioned downstream from the sample with respect to a propagation direction of the electron beam; and wherein the second detector is positioned upstream from the sample with respect to the propagation direction of the electron beam]. This detection channel may therefore be commonly referred to as a “top” channel [and wherein the second detector is positioned upstream from the sample with respect to the propagation direction of the electron beam], and the optical tool may also include two or more side channels [wherein the first detector is positioned downstream from the sample with respect to a propagation direction of the electron beam] configured as described above. As such, the optical tool may include at least three channels (i.e., one top channel and two side channels), and each of the at least three channels has its own collector, each of which is configured to collect light at different scattering angles than each of the other collectors.)
Regarding claim 12, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein the plurality of detectors includes a fourth detector for a third modality; (in [0113] Alignment 308 may be performed with any suitable non-learning based alignment or registration method known in the art such as NCC, sum square difference, etc. Therefore, the embodiments described herein can use a relatively simple alignment method to robustly align the images. In particular, images acquired with different modalities [wherein the plurality of detectors includes a fourth detector for a third modality as a fourth labeled detector in the plurality capturing a third modality/ ] (e.g., a SEM image and a trivially rendered design clip) often look very different from each other due to many factors such as optical proximity errors, missing layers in design (e.g., where a feature in the design (such as a liner) does not appear in an image of the specimen on which the design is formed), various types of noise in the specimen images, or difference in contrast between specimen images and design images… [0063] The system includes one or more computer subsystems, e.g., computer subsystem(s) 102 shown in FIG. 1, that may be configured for receiving the optical and electron beam images generated for the specimen. For example, as shown in FIG. 1, computer subsystem(s) 102 may be coupled to computer subsystem 36 and computer subsystem 124 such that computer subsystem(s) 102 can receive the optical images or output generated by detectors 28 and 34 and electron beam images or output generated by detector 134. Although the computer subsystem(s) may receive the optical images or output and the electron beam images or output from other computer subsystems coupled to the optical and electron beam tools, the computer subsystem(s) may he configured to receive the optical and electron beam images or output directly from the detectors that generate the images or output (e.g., if computer subsystems)102 are coupled directly to the detectors shown in FIG. 1)... [0065] The one or more virtual systems are not capable of having the specimen disposed therein. In particular, the virtual system(s) are not part of optical tool 10 or electron beam tool 122 and do not have any capability for handling the physical version of the specimen. In other words, in a system configured as a virtual system, the output of its one or more “detectors” may be output that was previously generated by one or more detectors of an actual tool and that is stored in the virtual system, and during the “imaging and/or scanning,” the virtual system may replay the stored output as though the specimen is being imaged and/or scanned [wherein the plurality of detectors includes a fourth detector for a third modality as the virtual detector of the replayed physical detector capturing a third modality]. In this manner, imaging and/or scanning the specimen with a virtual system may appear to be the same as though a physical specimen is being imaged and/or scanned with an actual system, while, in reality, the “imaging and/or scanning” involves simply replaying output for the specimen in the same manner as the specimen may be imaged and/or scanned.)
and wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective first input vector, the respective second input vector, and a respective third input vector to the respective (in [0074] The different modalities are different in at least one imaging parameter of at least one imaging system. In one embodiment, the first and second modalities generate the first and second images with different pixel sizes... In one such example, an image captured using an optical imaging system and an image captured using an electron beam imaging system are captured at different frequencies [and wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective first input vector, the respective second input vector, and a respective third input vector to the respective ]… [0091] Deep learning is part of a broader family of machine learning methods based on learning representations of data. An observation (e.g., an image) can be represented in many ways such as a vector of intensity values per pixel, or in a more abstract way as a set of edges, regions of particular shape, etc. Some representations are better than others at simplifying the learning task (e.g., face recognition or facial expression recognition)… [0110] As shown in FIG. 3, SEM image 300 (or a first image acquired for a specimen with a first modality) is input to learning based model 302, which transforms the SEM image to thereby render it into the common space of CAD image 306. In other words, learning based model 302 transforms SEM image 300 to rendered image 304 by mapping SEM image 300 from SEM image space to CAD image space [and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective ]... Since the rendered image and the CAD image now look as if they were acquired from the same modality prior to alignment, alignment can be performed relatively easily as described further herein. [0111] In the embodiment shown in FIG. 3, the learning based model may be a regression model or any of the learning based models described herein. In one such example, the learning based model may be in the form of a deep convolution autoencoder (DCAE) [and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective ]. The encoder portion of the learning based model may include, for example, five convolutional layers with kernel sizes of, for example, 5×5, a stride of 2, and no zero padding… And in [0115] The embodiment shown in FIG. 3 shows a run time mode of one of the alignment approaches described herein, FIG. 4 shows one possible method for training such an alignment approach. As shown in FIG. 4, the training may include inputting SEM image 400 into learning based model 402, which may be a regression model or another learning based model described herein. In this embodiment, the learning based model includes encoder 404 and decoder 408, Which may be configured as described further herein. In addition, although an auto-encoder is shown in this figure in the learning based model, any regression model such as CG AN or demise convolutional auto-encoder can be used in the embodiments described herein. Image 400 is input to encoder 404, which determines features 406 (i.e., learning or deep learning based features) [and wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective first input vector, the respective second input vector, and a respective third input vector to the respective ] of the image. Features 406 are input to decoder 408, which transforms the image into a different space. In this case, the decoder transforms the input SEM image from features 406 [and wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective first input vector, the respective second input vector, and a respective third input vector to the respective] to image 410 in design space. In this manner, image 410 may be a CAD image.) )
Ha teaches processing image data using machine learning models, including autoencoder models. One of ordinary skill in the art would understand that the latent space of an autoencoder can model corresponding features using a probability density function/distribution.
Hig expressly teaches that the latent space of an autoencoder can model corresponding features using a probability density function/distribution, in [0080] A variational autoencoder 151 is one type of autoencoder. An autoencoder [wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective first input vector, the respective second input vector, and a respective third input vector to the respective probability density in the latent space, the respective third input vector corresponding to the third modality] is a multilayer neural network that is created with machine learning such that input data and output data are identical to each other. The autoencoder compresses the input data into a vector having fewer dimensions than the input data, and restores the output data from the vector. Here, the variational autoencoder 151 is created such that a set of vectors follows a specific probability distribution [… wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective first input vector, the respective second input vector, and a respective third input vector to the respective probability density in the latent space, the respective third input vector corresponding to the third modality]. The variational autoencoder 151 includes an encoder 152 and a decoder 153. … [0082] A vector 155 is calculated between the encoder 152 and the decoder 153. The vector 155 is a representation of the features of the image 157 in low dimensions. For example, the vector 155 has 48 dimensions. The vector 155 may be called a latent variable, feature value, feature vector, or another. The vector 155 is mapped to a latent space 154. The latent space 154 is a vector space such as a 48-dimensional space. [0083] When a set of images of the same type (for example, a set of face photos or a set of handwritten characters) is input to the encoder 152, a set of vectors [… wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective first input vector, the respective second input vector, and a respective third input vector to the respective probability density in the latent space, the respective third input vector corresponding to the third modality ] corresponding to the set of images has a specific probability distribution such as a normal distribution in the latent space 154 […wherein the electronic controller is configured to, for each pixel of the base image, map, with the autoencoder, the respective … input vector, and a respective … input vector to the respective probability density in the latent space.. the respective… input vector corresponding to the … modality]. For example, the probability distribution in the latent space 154 is a multivariate normal distribution that has the vector 155 as a probability variable and that is specified by a specific mean vector and variance-covariance matrix. Here, a probability distribution other than the normal distribution may be assumed. The probability of occurrence of a specified vector in the set of vectors is approximated to a probability density calculated by a probability density function […and a respective … input vector to the respective probability density in the latent space..]. In general, a vector closer to the mean vector has a higher probability density, whereas a vector farther away from the mean vector has a lower probability density […and a respective … input vector to the respective probability density in the latent space..].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Sathish and Allen for the same reasons disclosed above.
Regarding claims 13 and 19, the claims have limitations are similar with claim 1 limitations and are rejected under the same rationales.
Regarding claim 20, the rejection of claim 19 is incorporated and Ha in combination with Hig teaches a non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising the method of claim 19. (in [0146] Each of the steps of the method may be performed as described further herein. The method may also include any other step(s) that can be performed by the system, computer system(s) or subsystem(s), imaging system(s), component(s), model(s), module(s), etc. described herein [a non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising the method of claim 19]. The one or more computer systems, the one or more components, and the model may be configured according to any of the embodiments described herein [a non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising the method of claim 19], e.g., computer subsystem(s) 102, component(s) 100, and model 104. In addition, the method described above may be performed by any of the system embodiments described herein. [0147] An additional embodiment relates to a non-transitory computer-readable medium storing program instructions executable on one or more computer systems for performing a computer-implemented method for aligning images for a specimen acquired with different modalities. One such embodiment is shown in FIG. 9. In particular, as shown in FIG. 9, non-transitory computer-readable medium 900 includes program instructions 902 executable on computer system(s) 904 [a non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising the method of claim 19]. The computer-implemented method may include any step(s) of any method(s) described herein.)
Claims 1, 13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ries et al. (US 20210232872, hereinafter ‘Ri’) in view of Warschauer (US 10714309, hereinafter ‘Wa’) in further view of Van Kraaij et al. (US 20250284191 hereinafter ‘Van’).
Regarding independent claim 1, Ri teaches an apparatus, comprising: an electron-beam column configured to scan an electron beam across a sample; a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality; (in [0047] In one embodiment, the SEM inspection sub-system 102b is configured to perform one or more measurements on the sample 120 [an electron-beam column configured to scan an electron beam across a sample]. In this regard, the SEM inspection sub-system 102b may be configured to acquire one or more images 125 of the sample 120 [the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality]. The one or more images 125 of the sample 120 may include one or more patch clips 135.. The SEM inspection sub-system 102b may include, but is not limited to, an electron beam source 128, one or more electron-optical elements 130, one or more electron-optical elements 132, and an electron detector assembly 134 including one or more electron sensors 136 [a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality]. [0050] For example, the system 100 may include one or more electron beam scanning elements (not shown)…)
and an electronic controller connected to receive streams of measurements from the plurality of detectors (in [0051] In another embodiment, secondary and/or backscattered electrons 131 are directed to one or more sensors 136 of the electron detector assembly 134. The electron detector assembly 134 of the SEM inspection sub-system 102b may include any electron detector assembly known in the art suitable for detecting backscattered and/or secondary electrons 131 emanating from the surface of the sample 120. In one embodiment, the electron detector assembly 134 includes an electron detector array [an electronic controller connected to receive streams of measurements from the plurality of detectors]. In this regard, the electron detector assembly 134 may include an array of electron-detecting portions. Further, each electron-detecting portion of the detector array of the electron detector assembly 134 may be positioned so as to detect an electron signal from sample 120 associated with one of the incident one or more electron beams 129 [an electronic controller connected to receive streams of measurements from the plurality of detectors]. The electron detector assembly 134 may include any type of electron detector known in the art....)
and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements, (in. [0061] It is noted herein that the system 100 may be configured to receive any size patch clip. For example, the system 100 may be configured to receive patch clips with a size of 64×64 (e.g., 4096 pixels) [pixel of a … image ; teaches patches is composed of pixels to be processed as patch clips]. By way of another example, the system may be configured to receive patch clips with a size of 256×256 (e.g., 65, 536 pixels)… [0088] In a step 304, one or more target clips are identified from one or more patch clips 135. The one or more images of the sample 120 may include the one or more patch clips 135 for every defect. [0089] In a step 306, one or more processed clips are prepared based on the one or more target clips. For example, one or more median clips may be generated based on the one or more target clips [each pixel of a base image of the sample generated] and one or more reference clips… For purposes of the present disclosure, the term “reference clips” refers to clips which are substantially clean (e.g., contain no defects). [0090] In a step 308, one or more encoded images are generated by transforming the one or more processed clips via an autoencoder [a base image of the sample generated using the imaging modality, map, with an autoencoder … and the base image being obtained based on the streams of measurements],. For example, the autoencoder may be configured to learn an effective low dimensional representation of the input data (e.g., the one or more patch clips) using a stacked autoencoder (e.g., a single three layered autoencoder) [for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector]. For instance, the autoencoder may receive the one or more processed clips from step 306 as an input (e.g., four 64×64 clips) and feed the one or more processed clips to a second layer (e.g., a convolutional neural network (CNN) layer). The second layer (e.g., the CNN layer) may include one or more filters.)
the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality; (in [0069] Typically, autoencoder training begins with randomly initialized weights and at every iteration, the loss function (Eqn. 2) drives the weights to reach an optimal solution. However, when the autoencoder has linear activation functions in encoder and decoder squared error loss function and normalized inputs the optimal weights are given by the eigenvectors [the base image being obtained based on the streams of measurements, the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality] of the covariance matrix of the input data, which may be described by Eqn. 3:... [0074] In an additional/alternative embodiment, the system 100 utilizes a principal component analysis (PCA) to encode the one or more encoded images [the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality]. For example, the PCA may compress the input data into the one or more encoded images. [0075] In another embodiment, in step 206, the system 100 is configured to sort the one or more encoded images into a set of clusters via a clustering algorithm. For example, once the autoencoder has the learned the mapping of the input data (as shown in step 204) the data is mapped and clustered using the clustering algorithm. For instance, similar objects may be grouped together into a manageable number of categories [the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality;] using the clustering algorithm…)
identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs; ([0075] In another embodiment, in step 206, the system 100 is configured to sort the one or more encoded images into a set of clusters via a clustering algorithm. For example, once the autoencoder has the learned the mapping of the input data (as shown in step 204) the data is mapped and clustered using the clustering algorithm)
and generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image. (in [0075] In another embodiment, in step 206, the system 100 is configured to sort the one or more encoded images into a set of clusters via a clustering algorithm. For example, once the autoencoder has the learned the mapping of the input data (as shown in step 204) the data is mapped and clustered using the clustering algorithm. For instance, similar objects may be grouped [generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image] together into a manageable number of categories using the clustering algorithm. In this regard, groups of objects are clustered together based on similar values in the one or more encoded images. The controller 104 may be configured to sort the one or more encoded images into a set of clusters using any clustering algorithm including, but not limited to, supervised learning, unsupervised learning, and the like.)
While Ri teaches the detector system for acquiring images associated with a sample for making developing a classification machine learning system/methods as noted above.
Ri does not expressly use the term modality for the captured images associated of an observation.
The Wa reference does expressly use the term modality for the captured images associated of an observation, in 2:30-45: Methods and systems for generating labeled images from a microscope detector by leveraging detector data from a different microscope detector of a different modality are disclosed. More specifically, the disclosure includes methods and systems for generating labeled electron/charged particle microscope images of a sample by utilizing detector systems of a different modality to generate labeled images of the sample [a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality], and then utilizing these labeled images to automatically label the images generated by the electron/charged particle microscope. In this way, the disclosed systems and methods automate the process of generating labeled electron microscopy images. This in turn greatly expedites the process of generating training sets for training a deep learning and/or neural network to analyze, label, and/or correct abnormalities in images obtained with electron microscopes (EM) and/or charged particle microscopes. And in 17:55-67: …The method of paragraphs B1-B8, further comprising: generating, using a third microscope detector system of a third modality and based on the emissions resultant from the focused charged beam being incident on the sample, third detector data of the third modality [a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality]; automatically generating, by the one or more processors, a third labeled image based on the third detector data of the third modality, wherein the generating the third labeled image comprises: generating, by the one or more processors and based on the third detector data, a third image of the sample; and determining, by the one or more processors and based on the third detector data, additional composition information about the portion of the sample…
Wa and Ri are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data captured from different detectors to automatically analyze and detect abnormalities, as disclosed by Wa with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as disclosed by Ri.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Wa and Ri as noted above; Doing so allows for developing information processing and modeling techniques to help identify defects in electron microscopy images using deep learning and to help drastically speed up this process, (Wa, 1:13-16).
While Wa and Ri teach the use of neural networks in processing and modeling information and the use of data processing using autoencoders as a tool for data dimensionality reduction, Wa and Ri do not expressly the use probability density modeling with autoencoders in data processing task.
Van teaches the use probability density modeling with autoencoders in data processing task, in [0102] In some embodiments, the low dimensional encoding z represents one or more features of an input… The one or more encoded features (dimensions) represented in the low dimensional encoding may be predetermined (e.g., by a programmer at the creation of the present modular autoencoder model) [map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space … identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs], determined by prior layers of a neural network, adjusted by a user via a user interface associated with a system described herein, and/or may be determined in by other methods. In some embodiments, a quantity of encoded features (dimensions) represented by the low dimensional encoding may be predetermined (e.g., by the programmer at the creation of the present modular autoencoder model),…And in [0108] The generating, the selecting, and the determining (e.g., operation 702, operation 704, and operation 706) are performed by an electronic model comprising an encoder structure and a generative structure (e.g., a decoder) with a conditional mapping sub-model [map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space … identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs]. In some embodiments, as described herein, the model is a machine learning model. In some embodiments, the model comprises encoder-decoder architecture. In an embodiment, the encoder-decoder architecture comprises variational encoder-decoder architecture, and operation 702 and/or other operations comprise training the variational encoder-decoder architecture with a probabilistic latent space, which generates realizations in an output space. The latent space comprises low dimensional encodings and/or other information (as described herein). A latent space is probabilistic if it is formed by sampling from a distribution (such as Gaussian) given the parameters of the distribution (such as mu and sigma) computed by the encoder [map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space … identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs]… [0117] Encoder structure 802 and generative structure 804 form a U-net type deep learning model 810. A real and continuous variational low-dimensional latent space 812 is included as part of model 800. During inference, an input image 814 (target design) is projected simultaneously to a CTM like image 816, as well as encoded into the latent-space, which models (via variational Bayes) the distribution of mask variants that can be generated. Given an input image 814, samples from a latent space probability density function will each generate their own mask variant 820 [map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space … identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs]. Since latent space 812 is variational, parameters such as σ.sub.prior give information about how varied output (SRAF+OPC) images 830 are for this particular input image 814.
Van, Wa and Ri are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing deep learning algorithms to be used to automatically model and identify processes in semiconductor processes as disclosed by Van with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as collectively disclosed by Wa and Ri.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Van, Wa and Ri as noted above; Doing so allows for developing information processing and modeling techniques to help improve the design and development of integrated circuits (Van, Abstract and 0003).
Regarding claims 13 and 19, the claims have limitations are similar with claim 1 limitations and are rejected under the same rationales.
Regarding claim 20, the rejection of claim 19 is incorporated and Ri in combination with Wa and Van teaches a non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising the method of claim 19. (in [0044] As noted previously herein, the controller 104 of system 100 may include one or more processors 106 and memory 108. The memory 108 may include program instructions configured to cause the one or more processors 106 to carry out various steps of the present disclosure. In one embodiment, the program instructions are configured to cause the one or more processors 106 to adjust one or more characteristics of the optical inspection sub-system 102a in order to perform one or more measurements of the sample 120… [0149] The memory 108 may include any storage medium known in the art suitable for storing program instructions executable by the associated one or more processors 106 and the data received from the inspection sub-system 102. For example, the memory 108 may include a non-transitory memory medium. For instance, the memory 108 may include, but is not limited to, a read-only memory (ROM), a random-access memory (RAM), a magnetic or optical memory device (e.g., disk), a magnetic tape, a solid-state drive and the like. It is further noted that memory 108 may be housed in a common controller housing with the one or more processors 106. In an alternative embodiment, the memory 108 may be located remotely with respect to the physical location of the processors 106, controller 104, and the like. In another embodiment, the memory 108 maintains program instructions for causing the one or more processors 106 to carry out the various steps described through the present disclosure.)
Claims 1-3, 13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bhaskar et al. (US 20170200265, hereinafter ‘Has’) in view of Qu et al. (US 20230267600 hereinafter ‘Qu’) in further view of Warschauer (US 10714309, hereinafter ‘Wa’).
Regarding independent claim 1, Has teaches an apparatus, comprising: an electron-beam column configured to scan an electron beam across a sample; a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality; and an electronic controller connected to receive streams of measurements from the plurality of detectors (in [0102] In an additional embodiment, the information for the specimen includes two or more actual electron beam images of the specimen [an apparatus, comprising: an electron-beam column configured to scan an electron beam across a sample; a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample], and the two or more actual electron beam images include two or more actual electron beam images corresponding to different values of a parameter of an electron beam tool,…; And in [0030] In some instances, the optical tool may be configured to direct light to the specimen at more than one angle of incidence at the same time. For example, the illumination subsystem may include more than one illumination channel, one of the illumination channels [the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality] may include light source 16, optical element 18, and lens 20 as shown in FIG. 1 ... [0039] The one or more detection channels may include any suitable detectors [the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality] known in the art. For example, the detectors [the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality] may include photo-multiplier tubes (PMTs), charge coupled devices (CCDs), time delay integration (TDI) cameras, and any other suitable detectors known in the ail. The detectors may also include non-imaging detectors or imaging detectors. In this manner, if the detectors are non-imaging detectors, each of the detectors may be configured to detect certain characteristics of the scattered light [and an electronic controller connected to receive streams of measurements from the plurality of detectors] such as intensity but may not be configured to detect such characteristics as a function of position within the imaging plane. As such, the output that is generated by each of the detectors included in each of the detection channels of the optical tool may be signals or data, but not image signals or image data. In such instances, a computer subsystem such as computer subsystem 36 may be configured to generate images of the specimen from the non-imaging output of the detectors [and an electronic controller connected to receive streams of measurements from the plurality of detectors]. However, in other instances, the detectors may be configured as imaging detectors that are configured to generate imaging signals or image data [and an electronic controller connected to receive streams of measurements from the plurality of detectors]. Therefore, the optical tool may be configured to generate optical images described herein in a number of ways.)
and configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements, the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality; (in [0089] An optical image stack (i.e., multiple optical images) is represented herein by the notation O(x, y, p1, p2, pn), which is the pixel value at (x, y) location observed on an optical tool under the optical and process parameters (p1, p2, . . . , pn) [configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements]. Similarly, an electron beam image stack (i.e., multiple electron beam images) is represented herein by the notation S(x, y, p1, p2, . . . , pn), which is the pixel value at (x, y) location observed on an electron beam tool under the electron beam and process parameters of (p1, p2, . . . , pn) [configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements]. Design information is represented herein by the notation D(x, y, l), which is the pixel value at (x, y) location of design information, e.g., a rendered design, for the 1 -th wafer layer. Specimen features are represented herein by the notation W(x, y), which refers to certain features at (x, y) location of a specimen [the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality], e.g., defect class information, patterned features, electrical test results, etc. These features are either computable from subsets of design, optical images, and electron beam images via currently used algorithms or directly measureable from a semiconductor tool. [0090] .., the learning based model is expected to predict a subset of representations under a fixed set of the same or different parameters [the respective first input vector corresponding to the first modality, the respective second input vector corresponding to the second modality]. Several such examples will now be described… [0099] In one embodiment, the learning based model is configured for mapping [configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements] a relationship between patterned features on the specimen and one or more of the optical images, the electron beam images, and the design data. For example, as described further herein, the specimen features may be defined as W(x, y) referring to certain features at (x, y) locations on a specimen. In this manner, as described further herein, given one or more of an optical image, an electron beam image, design, and/or features on a specimen as input collected with a fixed set of imaging and/or process parameters, the learning based models described herein can predict a subset of representations under the same or different parameters.
Examiner notes the learning model includes an autoencoder in [0137] In some embodiments, the learning based model includes a generative model. A “generative” model can be generally defined as a model that is probabilistic in nature… Instead, as described further herein, the generative model can be learned (in that its parameters can be learned) based on a suitable training set of data. [0138] In one such embodiment, the generative model includes an autoencoder variant, a generative adversarial network, a conditional generative adversarial network, or a deep generative model. For example, for learning a transformation under fixed imaging or process parameters as described above, the learning based model may be configured for a generative approach using one or more generative models including autoencoder variations [configured to: for each pixel of a base image of the sample generated using the imaging modality, map, with an autoencoder, a respective first input vector and a respective second input vector to a respective probability density in a latent space, with the respective first input vector, the respective second input vector, and the base image being obtained based on the streams of measurements], in which the decoder part will eventually be used for representation conversion…)
and generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image. (in [0140] An autoencoder, autoassociator or Diabolo network is an artificial neural network used for unsupervised learning of efficient codings. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction. Recently, the autoencoder concept has become more widely used for learning generative models of data. Architecturally, the simplest form of an autoencoder is a feedforward, non-recurrent neural network very similar to the multilayer perceptron (MLP)—having an input layer, an output layer and one or more hidden layers connecting them—, but with the output layer having the same number of nodes as the input layer, and with the purpose of reconstructing its own inputs (instead of predicting the target value given inputs). Therefore, autoencoders are unsupervised learning models. An autoencoder always consists of two parts, the encoder and the decoder. Various techniques exist, to prevent autoencoders from learning the identity function and to improve their ability to capture important information and learn richer representations [generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image]. The autoencoder may include any suitable variant of autoencoder such as a Denoising autoencoder, sparse autoencoder, variational autoencoder, and contractive autoencoder.)
While Has teaches the use of neural networks in processing and modeling information and the use of data processing using autoencoders. Has do not expressly the use probability density modeling with autoencoders in data processing task as claimed in the limitation:
identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs;
Qu does expressly teach the use probability density modeling with autoencoders in data processing task as claimed in the limitation:
identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs; (in [0006] Aspects of example embodiments of the present disclosure relate to vector quantized auto-encoder codebook learning [identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belong] for manufacturing display extreme minor defects detection… [0010] In one or more embodiments, a code from among the codes in the assigned code matrix is assigned to an image patch of the plurality of image patches of the input image based on a vector quantization of a latent feature corresponding to the image patch [respective latent-space cluster]. In one or more embodiments, a vector quantization loss of the system includes a reconstruction loss that occurs during generation of the reconstructed image and a loss that occurs during the vector quantization of latent features corresponding to the plurality of image patches of the input image… [0013] In one or more embodiments, a code from the codebook that is of a shortest distance to the latent feature vector of an image patch of the plurality of image patches among the codes in the codebook is assigned to the image patch. In one or more embodiments, the anomaly score of each of the assigned codes of the patch-set [respective latent-space cluster] is determined based on a probability density function [identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belong].)
Additionally, Qu teaches and generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image in [0071] In one or more embodiments, each of FIGS. 2A, 2C, and 2E illustrates an input image (e.g., the input image (x) 110 of FIG. 1) and each of FIGS. 2B, 2D, and 2F illustrates a reconstructed image [a cluster-mapped image of the sample] (e.g., the reconstructed image [generate a cluster-mapped image of the sample based on the base image and further based on latent-space clusters identified for different pixels of the base image] ({circumflex over (x)}) 170 of FIG. 1) generated from the input image of FIG. 2A, by the PVQAE system 100 of FIG. 1. [0072] In one or more embodiments, real-world images from production line may incorporate sophisticated visual patterns that contain subtle defective traits. In order to learn an effective codebook from such complex datasets, it may be desirable to adopt a model that can concurrently (e.g., simultaneously) learn local features and understand the global composition of these local features. By encoding the information of both perspectives, the product image can be represented by a series of locally vivid as well as globally coherent perceptional codes [… further based on latent-space clusters identified for different pixels of the base image]. In order to achieve this, on top of convolutional layers, the multi-head self-attention layer may be adopted to learn the inter-correlation dependencies between elements within a sequence (i.e. words for language tasks or image patches for vision tasks […based on the base image and further based on latent-space clusters identified for different pixels of the base image]). As a consequence, the PVQAE system 100 can learn a codebook with richer perceptions of complex industrial products. [0073] In one or more embodiments, PVQAE system 100 may also determine defects in a system. For example, FIG. 3 illustrates defect detection using patch-wise codebook learning […. based on the base image and further based on latent-space clusters identified for different pixels of the base image]. [0074] For example, given an input image x at test, the trained PVQAE system 100 may identify defects by estimating the anomaly score s.sub.i,j for each image patch (e.g., 310(1), 310(2), 310(30) . . . , 310(n)) indexed at i, j from the assigned codes z.sub.q.
Qu and Has are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing learning algorithms to be used to automatically analyze abnormalities in images as disclosed by Qu with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as disclosed by Has.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Qu and Has as noted above; Doing so allows for developing information processing and modeling techniques to achieve automated visual defect detection and to enhance equipment and quality control in a defect detection system, (Qu, 0038-0039).
Qu and Has teach the information and modeling processing techniques for developing a detection system using machine learning models and methods.
Additionally, Wa teaches the collection of detector data for processing multi-modal data for developing automated detection systems.
The Wa reference does expressly use the term modality for the captured images associated of an observation, in 2:30-45: Methods and systems for generating labeled images from a microscope detector by leveraging detector data from a different microscope detector of a different modality are disclosed. More specifically, the disclosure includes methods and systems for generating labeled electron/charged particle microscope images of a sample by utilizing detector systems of a different modality to generate labeled images of the sample [a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality], and then utilizing these labeled images to automatically label the images generated by the electron/charged particle microscope. In this way, the disclosed systems and methods automate the process of generating labeled electron microscopy images. This in turn greatly expedites the process of generating training sets for training a deep learning and/or neural network to analyze, label, and/or correct abnormalities in images obtained with electron microscopes (EM) and/or charged particle microscopes. And in 17:55-67: …The method of paragraphs B1-B8, further comprising: generating, using a third microscope detector system of a third modality and based on the emissions resultant from the focused charged beam being incident on the sample, third detector data of the third modality [a plurality of detectors configured to measure signals caused by interaction of the electron beam with the sample, the plurality of detectors including a first detector for a first modality, a second detector for a second modality, and a third detector for an imaging modality]; automatically generating, by the one or more processors, a third labeled image based on the third detector data of the third modality, wherein the generating the third labeled image comprises: generating, by the one or more processors and based on the third detector data, a third image of the sample; and determining, by the one or more processors and based on the third detector data, additional composition information about the portion of the sample…
Wa, Qu and Has are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data captured from different detectors to automatically analyze and detect abnormalities, as disclosed by Wa with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as collectively disclosed by Qu and Has.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Wa, Qu and Has as noted above; Doing so allows for developing information processing and modeling techniques to help identify defects in electron microscopy images using deep learning and to help drastically speed up this process, (Wa, 1:13-16).
Regarding claim 2, the rejection of claim 1 is incorporated and Has in combination with Qu and Wa teaches the apparatus of claim 1, wherein the plurality of detectors is selected from the group consisting of a high-angle annular dark field detector, a medium-angle annular dark field detector, an annular bright field detector, a segmented annular detector, a differential phase contrast detector, an Electron Energy Loss Spectroscopy (EELS) detector, an Energy-Dispersive X-ray Spectroscopy (EDS) detector, and a two-dimensional diffraction-pattern detector. (in [0041] Computer subsystem 36 coupled to the optical tool may be coupled to the detectors [the plurality of detectors] of the optical tool in any suitable mariner (e,g., via one or more transmission media, which may include “wired” and/or “wireless” transmission media) such that the computer subsystem can receive the output generated by the detectors for the specimen. Computer subsystem 36 may be configured to perform a number of functions described further herein using the output of the detectors. And [0038] As described further above, each of the detection channels included in the optical tool may be configured to detect scattered light. Therefore, the optical tool shown in FIG. 1 may be configured for dark field (DF) imaging of specimens [a high-angle annular dark field detector, a medium-angle annular dark field detector]. However, the optical tool may also or alternatively include detection channel(s) that are configured for bright field (BF) [an annular bright field detector]… [0055] The systems described herein may also include one or more additional tools configured to generate other output for the specimen such as an ion beam-based tool. Such a tool may be configured as shown in FIG. 1 with respect to the electron beam tool except that the electron beam source may be replaced with any suitable ion beam source known in the art. In addition, the tool may be any other suitable ion beam tool such as those included in commercially available focused ion beam (FIB) systems, helium ion microscopy (HIM) systems, and secondary ion mass spectroscopy (SIMS) systems [an Electron Energy Loss Spectroscopy (EELS) detector].)
Wa teaches the plurality of detectors is selected from the group consisting of a high-angle annular dark field detector, a medium-angle annular dark field detector, an annular bright field detector, a segmented annular detector, a differential phase contrast detector, an Electron Energy Loss Spectroscopy (EELS) detector, an Energy-Dispersive X-ray Spectroscopy (EDS) detector, and a two-dimensional diffraction-pattern detector. (in 3:42-4:10: In FIG. 1, the first microscope detector system 102 is illustrated as being a disk-shaped bright field detector 134. In some embodiments, the first microscope detector system 102 may include one or more other detectors, e.g., a dark field detector [a high-angle annular dark field detector,]. In such embodiments, EM and/or charged particle microscope setup(s) 100 may simultaneously detect signals from one or more of the bright field detector 134 [an annular bright field detector] and the dark field detector 136 [a high-angle annular dark field detector, a medium-angle annular dark field detector]. Alternatively, or in addition, the first microscope detector system 102 may include a scanning electron microscope detector system, a focused ion beam detector system, a scanning electron microscope secondary electron detector system, a focused ion beam secondary electron detector system, and an optical microscope detector system… For example, FIG. 1 illustrates the second microscope detector system 104 as being a dispersion X-ray detector. In other embodiments, the second microscope detector system 104 may correspond to one or more of a high angle dark field detector system [a high-angle annular dark field detector], a dispersion x-ray detector system [an Energy-Dispersive X-ray Spectroscopy (EDS) detector], a back scatter detector system, an electron energy loss spectroscopy detector system [an Energy-Dispersive X-ray Spectroscopy (EDS) detector], a secondary ion detector system, and a secondary ion mass spectroscopy detector system... FIG. 2 shows example EM and/or charged particle microscope setup(s) 100 as being a scanning electron microscope with energy dispersive X-ray spectroscopy (SEM/EDX) system [an Electron Energy Loss Spectroscopy (EELS) detector] 150 for generating labeled images from a first microscope detector system 102 by leveraging detector data from a second microscope detector system 104 of a different modality. )
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Wa, Qu and Has for the same reasons disclosed above.
Regarding claim 3, the rejection of claim 1 is incorporated and Has in combination with Qu and Wa teaches the apparatus of claim 1, wherein the first detector is an EELS detector; and wherein the second detector is a(in [0041] Computer subsystem 36 coupled to the optical tool may be coupled to the detectors [the plurality of detectors including wherein the first detector is a] of the optical tool in any suitable mariner (e,g., via one or more transmission media, which may include “wired” and/or “wireless” transmission media) such that the computer subsystem can receive the output generated by the detectors for the specimen… [0055] The systems described herein may also include one or more additional tools configured to generate other output for the specimen such as an ion beam-based tool. Such a tool may be configured as shown in FIG. 1 with respect to the electron beam tool except that the electron beam source may be replaced with any suitable ion beam source known in the art. In addition, the tool may be any other suitable ion beam tool such as those included in commercially available focused ion beam (FIB) systems, helium ion microscopy (HIM) systems, and secondary ion mass spectroscopy (SIMS) systems [wherein the first detector is an EELS detector].)
Wa teaches wherein the first detector is an EELS detector; and wherein the second detector is an EDS detector. (in 3:42-4:10: In FIG. 1, the first microscope detector system 102 is illustrated as being a disk-shaped bright field detector 134. In some embodiments, the first microscope detector system 102 may include one or more other detectors, e.g., a dark field detector. In such embodiments, EM and/or charged particle microscope setup(s) 100 may simultaneously detect signals from one or more of the bright field detector 134 and the dark field detector 136. Alternatively, or in addition, the first microscope detector system 102 may include a scanning electron microscope detector system, a focused ion beam detector system, a scanning electron microscope secondary electron detector system, a focused ion beam secondary electron detector system, and an optical microscope detector system… For example, FIG. 1 illustrates the second microscope detector system 104 as being a dispersion X-ray detector. In other embodiments, the second microscope detector system 104 may correspond to one or more of a high angle dark field detector system, a dispersion x-ray detector system [wherein the second detector is an EDS detector], a back scatter detector system, an electron energy loss spectroscopy detector system [wherein the second detector is an EDS detector], a secondary ion detector system, and a secondary ion mass spectroscopy detector system... FIG. 2 shows example EM and/or charged particle microscope setup(s) 100 as being a scanning electron microscope with energy dispersive X-ray spectroscopy (SEM/EDX) system [wherein the first detector is an EELS detector] 150 for generating labeled images from a first microscope detector system 102 by leveraging detector data from a second microscope detector system 104 of a different modality. )
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Wa, Qu and Has for the same reasons disclosed above.
Regarding claims 13 and 19, the claims have limitations are similar with claim 1 limitations and are rejected under the same rationales.
Regarding claim 20, the rejection of claim 19 is incorporated and Has in combination with Qu and Wa teaches a non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising the method of claim 19. (in [0042] The computer subsystems shown in FIG. 1 (as well as other computer subsystems described herein) may also be referred to herein as computer system(s). Each of the computer subsystem(s) or system(s) described herein may take various terms, including a personal computer system, image computer, mainframe computer system, workstation, network appliance, Internet appliance, or other device. In general, the term “computer system” may be broadly defined to encompass any device having one or more processors, which executes instructions from a memory medium. The computer subsystem(s) or system(s) may also include any suitable processor known in the art such as a parallel processor. In addition, the computer subsystem(s) or system(s) may include a computer is platform with high speed processing and software, either as a standalone or a networked tool… [0165] An additional embodiment relates to a non-transitory computer-readable medium storing program instructions executable on one or more computer systems for performing a computer-implemented method for generating simulated output for a specimen. One such embodiment is shown in FIG. 3. In particular, as shown in FIG. 3, non-transitory computer-readable medium 300 includes program instructions 302 executable on computer system(s) 304. The computer-implemented method may include any step(s) of any method(s) described herein.)
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (US 20180330511, hereinafter ‘Ha’) in view of Higuchi et al. (US 20220301288, hereinafter ‘Hig’) in view of Besenbruch et al (US 20230154055, hereinafter ‘Bes’).
Regarding claim 4, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein the autoencoder comprises: … and wherein the neural network encoder and the neural network decoder have been trained using a loss function, the training spectra, and the reconstructed spectra, … (in [0116] Image 410 may be compared to a CAD image known to correspond to image 400. For example, image 410 may be compared to image 412, which may be the CAD image known to correspond to SEM image 400, L2-loss [wherein the neural network encoder and the neural network decoder have been trained using a loss function, the training spectra, and the reconstructed spectra,] step 414 may then determine differences between the two images, and the differences will be due to errors in parameters of the learning based model. In this manner, minimize step 416 may be performed to minimize the L2-loss step results thereby minimizing errors in the learning based model results and the parameters of the learning based model…)
Hig further teaches the apparatus of claim 1, wherein the autoencoder comprises: a neural network encoder configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in the latent space; and a neural network decoder configured to generate reconstructed spectra based on mappings, with the neural network encoder, of training spectra to the latent space; (As depicted in Fig. 4 and in [0079] FIG. 4 illustrates an example of a variational autoencoder. [0080] A variational autoencoder 151 is one type of autoencoder... Here, the variational autoencoder 151 is created such that a set of vectors follows a specific probability distribution [wherein the autoencoder comprises: a neural network encoder configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in the latent space]. The variational autoencoder 151 includes an encoder 152 [wherein the autoencoder comprises: a neural network encoder configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in the latent space] and a decoder 153 [and a neural network decoder configured to generate reconstructed spectra based on mappings, with the neural network encoder, of training spectra to the latent space]. [0081] The encoder 152 receives an image 157 as an input. The encoder 152 is a multilayer neural network with a plurality of layers. An output from the encoder 152 has fewer dimensions than the input to the encoder 152… For example, in the layers of the encoder 152, the number of dimensions decreases stepwise in the direction from the input layer toward the output layer. The decoder 153 outputs an image 158 [a neural network decoder configured to generate reconstructed spectra based on mappings, with the neural network encoder, of training spectra to the latent space]. The height and width of the image 158 are identical to those of the image 157. The image 158 is ideally identical to the image 157. The decoder 153 is a multilayer neural network with a plurality of layers. An output from the decoder 153 has more dimensions than an input to the decoder 153 [a neural network decoder configured to generate reconstructed spectra based on mappings, with the neural network encoder, of training spectra to the latent space]. For example, in the layers of the decoder 153, the number of dimensions increases stepwise in the direction from the input layer toward the output layer. [0082] A vector 155 is calculated between the encoder 152 and the decoder 153. The vector 155 is a representation of the features of the image 157 in low dimensions. For example, the vector 155 has 48 dimensions. The vector 155 may be called a latent variable, feature value, feature vector, or another. The vector 155 is mapped to a latent space 154. The latent space 154 is a vector space such as a 48-dimensional space. [0083] When a set of images of the same type (for example, a set of face photos or a set of handwritten characters) is input to the encoder 152, a set of vectors corresponding to the set of images has a specific probability distribution such as a normal distribution in the latent space 154 [wherein the autoencoder comprises: a neural network encoder configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in the latent space]…The probability of occurrence of a specified vector in the set of vectors is approximated to a probability density calculated by a probability density function [wherein the autoencoder comprises: a neural network encoder configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in the latent space]. In general, a vector closer to the mean vector has a higher probability density, whereas a vector farther away from the mean vector has a lower probability density. [0084] To make the vector 155 follow the specific probability distribution, the encoder 152 calculates a value representing the mean vector (p) and a value representing the variance-covariance matrix (E) for the input image 157. Sampling for extracting the vector 155 is performed from the probability distribution specified by the values calculated by the encoder 152...)
While Ha in combination with Hig teaches the use of autoencoders in modeling data distributions over a training image spectra as noted above.
Ha and Hig do not expressly disclose the loss associated with an autoencoder model includes claimed terms.
Bes does expressly teach the loss associated with an autoencoder model includes claimed terms, in [0564] The loss function within learnt compression can in its simplest form be considered to be composed of two different terms [wherein the neural network encoder and the neural network decoder have been trained using a loss function, the training spectra, and the reconstructed spectra, the loss function including a sum of a term representing reconstruction loss and a regularizer term]: one term that controls the distortion of the compressed image [term representing reconstruction loss] or video, D, and another term [and a regularizer term] that controls the size of the compressed media (rate) R which is typically measured as the number of bits required per pixel (bpp). An uncompressed image requires 24 bpp, most compressed images are below 0.5 bpp…
PNG
media_image1.png
46
134
media_image1.png
Greyscale
And in [0590] In AI-based data compression, an autoencoder is a class of neural network whose parameters are tuned, in training, primarily to perform the following two tasks jointly: [0591] 1. Find a compressed latent representation of the input data such that the description of that representation is as short as possible; [0592] 2. Given the latent representation of the data, transform it back into its input either exactly (lossless compression) or approximately (lossy compression). [0593] Here we assume a lossy compression pipeline, however it should be noted that many concepts presented here are also applicable in lossless compression. The aforementioned tasks form the framework of a joint optimisation problem of two loss terms [wherein the neural network encoder and the neural network decoder have been trained using a loss function, the training spectra, and the reconstructed spectra, the loss function including a sum of a term representing reconstruction loss and a regularizer term] commonly found in compression problems, namely the minimisation of metrics representing rate, R(⋅), and distortion, D(⋅), respectively [the loss function including a sum of a term representing reconstruction loss and a regularizer term]. The rate-distortion minimization objective can mathematically be expressed in form of a weighted sum [the loss function including a sum of a term representing reconstruction loss and a regularizer term] denoted by …
PNG
media_image2.png
54
274
media_image2.png
Greyscale
where x is the input data, θ is the network parameters and λ is a weighting factor that controls the rate-distortion balance…
Bes, Hig and Ha are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data for developing methods and systems for image compression and decoding using neural network models, as disclosed by Bes with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as collectively disclosed by Hig and Ha.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Bes, Hig and Ha as noted above; Doing so allows for developing information processing and modeling techniques to recreate high-quality approximation of the desired image and reduce the amount of data required to transfer an image of a given quality in image processing systems, (Bes, 0005).
Regarding claim 14, the limitations are similar with claim 4 limitations and are rejected under the same rationales.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (US 20180330511, hereinafter ‘Ha’) in view of Higuchi et al. (US 20220301288, hereinafter ‘Hig’) in view of Van Rozendaal et al. (US 20220103839, hereinafter ‘Roz’).
Regarding claim 5, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein the autoencoder comprises: a first neural network encoder configured to map the respective first input vector to a first probability density in a first private subspace of the latent space; and a second neural network encoder configured to map the respective second input vector to a second probability density in a second private subspace of the latent space; and wherein the first neural network encoder and the second neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space.. (in [0122] One such embodiment is shown in FIG. 5. As shown in this embodiment, learning based model 500 includes encoders 502 and 506. First image 504 generated by a first modality, in this instance SEM, is input to encoder 502 [wherein the autoencoder comprises: a first neural network encoder configured to map the respective first input vector to a first probability density in a first private subspace of the latent space] while second image 508 generated by a second modality different from the first, in this instance CAD, in input to encoder 506 [and a second neural network encoder configured to map the respective second input vector to a second probability density in a second private subspace of the latent space]. Encoder 502 generates learning based model features(not shown) of image 504 while encoder 506 determines learning based model features (not shown) of image 508. The deep learning based features of the first and second images are input to concatenation (or “concat”) layer 510 [and wherein the first neural network encoder and the second neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space] of the learning based model…)
Ha does not expressly teach the use of a density function for modeling the feature space associated with an encoder model.
Hig expressly teaches the use of a density function for modeling the feature space associated with an encoder model. (in [0080] A variational autoencoder 151 is one type of autoencoder. An autoencoder is a multilayer neural network that is created with machine learning such that input data and output data are identical to each other. The autoencoder compresses the input data into a vector having fewer dimensions than the input data, and restores the output data from the vector. Here, the variational autoencoder 151 is created such that a set of vectors follows a specific probability distribution [… wherein the autoencoder comprises: …neural network encoder configured to map the respective first input vector to a first probability density in a ... private subspace of the latent space; …]. The variational autoencoder 151 includes an encoder 152 and a decoder 153. … [0082] A vector 155 is calculated between the encoder 152 and the decoder 153. The vector 155 is a representation of the features of the image 157 in low dimensions. For example, the vector 155 has 48 dimensions. The vector 155 may be called a latent variable, feature value, feature vector, or another. The vector 155 is mapped to a latent space 154. The latent space 154 is a vector space such as a 48-dimensional space. [0083] When a set of images of the same type (for example, a set of face photos or a set of handwritten characters) is input to the encoder 152, a set of vectors […neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space. ] corresponding to the set of images has a specific probability distribution such as a normal distribution in the latent space 154. For example, the probability distribution in the latent space 154 is a multivariate normal distribution that has the vector 155 as a probability variable and that is specified by a specific mean vector and variance-covariance matrix. Here, a probability distribution other than the normal distribution may be assumed. The probability of occurrence of a specified vector in the set of vectors is approximated to a probability density calculated by a probability density function. In general, a vector closer to the mean vector has a higher probability density, whereas a vector farther away from the mean vector has a lower probability density...)
While Ha in combination with Hig teaches the use of autoencoders in modeling using probability density functions to cluster vectors.
Ha and Hig do not expressly disclose expressly disclosed the use of a shared knowledge space.
Roz does expressly disclosed the use of a shared knowledge space, in [0138] FIG. 6 is a diagram illustrating an example neural network compression system 600 for instance-adaptive data compression. The neural network compression system 600 can be trained and further fine-tuned for the data being compressed to provide compression adapted/fine-tuned to that data being compressed (e.g., instance adapted). In this example, the neural network compression system 600 is shown implementing a mean-scale hyperprior model architecture using a variable autoencoder (VAE) framework. In some cases, a shared hyperdecoder [… wherein the first neural network encoder and the second neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space] can be used to predict the mean and scale parameters for the mean-scale hyperprior model. [0139] As shown in FIG. 6, the neural network compression system 600 can be trained using a training dataset 602. The training dataset 602 can be processed by an encoder 606 of a codec 604 to generate a latent space representation 608 (z.sub.2) [wherein the autoencoder comprises: a first neural network encoder configured to map the respective first input vector to a first probability density in a first private subspace of the latent space] of the training dataset 602. The encoder 606 can provide the latent space representation 608 (z.sub.2) to a decoder 610 of the codec 604 and a hyperencoder 614 of a hypercodec 612. [0140] The hyperencoder 614 [and a second neural network encoder configured to map the respective second input vector to a second probability density in a second private subspace of the latent space;] can use the latent space representation 608 (z.sub.2) and a latent prior 622 of a hyperprior 620 to generate a hyperlatent space representation 616 (z.sub.1) [and wherein the first neural network encoder and the second neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space] of the training dataset 602. In some examples, the hyperlatent space representation 616 (z.sub.1) and the hyperlatent space representation 616 (z.sub.1) can provide a hierarchical latent variable model for latent space z={z.sub.1, z.sub.2}. [0141] A hyperdecoder 618 of the hypercodec 612 can use the hyperlatent space representation 616 (z.sub.1) to generate a hyperprior model 624. The hyperdecoder 618 can predict mean and scale parameters for the hyperprior model 624. In some examples, the hyperprior model 624 can include a probability distribution over the parameters of the latent space representation 608 [wherein the autoencoder comprises: a first neural network encoder configured to map the respective first input vector to a first probability density in a first private subspace of the latent spac] (z.sub.2) and the hyperlatent space representation 616 (z.sub.1) [and a second neural network encoder configured to map the respective second input vector to a second probability density in a second private subspace of the latent space;]. In some examples, the hyperprior model 624 can include a probability distribution over the parameters [and wherein the first neural network encoder and the second neural network encoder are further configured to jointly map the respective first input vector and the respective second input vector to the respective probability density in a shared subspace of the latent space] of the latent space representation 608 (z.sub.2), the hyperlatent space representation 616 (z.sub.1), and the hyperdecoder 618.
Roz, Hig and Ha are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data for developing methods and systems using machine learning systems to compress image, as disclosed by Roz with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as collectively disclosed by Hig and Ha.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Roz, Hig and Ha as noted above; Doing so allows for developing information processing and modeling techniques to encode compressed data into a form that uses a lower bit rate, while avoiding or minimizing degradations in the compressed data quality, (Roz, 0003).
Regarding claim 15, the limitations are similar with claim 5 limitations and are rejected under the same rationales.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (US 20180330511, hereinafter ‘Ha’) in view of Higuchi et al. (US 20220301288, hereinafter ‘Hig’) in view of Van Helfenstein et al. (US 20250021020, hereinafter ‘Helf’).
Regarding claim 6, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein each of the respective first input vector and the respective second input vector has a respective dimensionality… (in [0007] Metrology processes are also used at various steps during a semiconductor manufacturing process to monitor and control the process. Metrology processes are different than inspection processes in that, unlike inspection processes in which defects are detected on specimens, metrology processes are used to measure one or more characteristics of the specimens that cannot be determined using currently used inspection tools. For example, metrology processes are used to measure one or more characteristics of specimens such as a dimension (e.g., line width, thickness, etc.) of features [wherein each of the respective first input vector and the respective second input vector has a respective dimensionality…] formed on the specimens during a process such that the performance of the process can be determined from the one or more characteristics… And in [0074] The different modalities are different in at least one imaging parameter of at least one imaging system. In one embodiment, the first and second modalities generate the first and second images with different pixel sizes [wherein each of the respective first input vector and the respective second input vector has a respective dimensionality…]... In one such example, an image captured using an optical imaging system and an image captured using an electron beam imaging system are captured at different frequencies [wherein each of the respective first input vector and the respective second input vector has a respective dimensionality…]… For example, the first and second images may be acquired with different wavelength ranges (or frequency spreads as wavelength and frequency are closely related) [wherein each of the respective first input vector and the respective second input vector has a respective dimensionality…] of beams. In one such example, an image captured using an optical imaging system and an image captured using an electron beam imaging system are captured at different frequencies.; And in [0123] In embodiments described herein in which the feature space is used as the common space for image alignment or registration, the feature space of each imaging modality can be different [wherein each of the respective first input vector and the respective second input vector has a respective dimensionality…]. It is driven by the data that is used to train the model. The training process will determine what are the best features to describe the images from each image modality (e.g., by minimizing the cost functions). Specifically, the deep features of the first image and the deep features of the second image are two output column vectors from the two encoders shown in FIG. 5. The two feature vectors do not need to have the same dimensions [wherein each of the respective first input vector and the respective second input vector has a respective dimensionality…]. Also, meanings of elements in each feature vector may be totally different. They are driven by data through the training process… )
and wherein the latent space is a two-dimensional space, a four-dimensional space, or an eight-dimensional space. (in [0101] An autoencoder, autoassociator or Diabolo network is an artificial neural network used for unsupervised learning of efficient codings. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction [wherein the latent space is a two-dimensional space, a four-dimensional space, or an eight-dimensional space]... An autoencoder always consists of two parts, the encoder and the decoder. Various techniques exist to prevent autoencoders from learning the identity function and to improve their ability to capture important information and learn richer representations. The autoencoder may include any suitable variant of autoencoder such as a Denoising autoencoder, sparse autoencoder, variational autoencoder, and contractive autoencoder..)
While Ha teaches processing multi-dimensional input data, Ha does not expressly teach encoding a dimensional vector input as claimed input vector has a respective dimensionality a range between 100 and 10000.
Hig expressly teaches encoding a dimensional vector input as claimed input vector has a respective dimensionality a range between 100 and 10000. (in [0063] The image 142 is a set of pixel values arranged in a grid. The pixel values are numerical values that each indicate the luminance of a pixel. The image 142 is represented as a tensor, which is a multidimensional array. In the case where the image 142 is a monochrome image, the image 142 is represented as a binary tensor in two-dimensional array having a predetermined height and a predetermined width. In the case where the image 142 is a color image, the image 142 is represented as a ternary tensor in three-dimensional array having three channels corresponding to a predetermined height, a predetermined width, and red-green-blue (RGB). The height and width of the image 142 to be input to the classification model 141 are adjusted to a predetermined height and width in advance. More specifically, the number of pixel values to be input to the classification model 141 is equal to the value calculated by “height×width×the number of channels.” This value calculated by “height×width×the number of channels” indicates the number of dimensions for the input [input vector has a respective dimensionality a range between 100 and 10000]. )
Alternatively, Helf expressly expressly teaches encoding a dimensional vector input as claimed input vector has a respective dimensionality a range between 100 and 10000. (in [0105] In one implementation of step 602, the sets of measured signals in the dataset (e.g. the diffraction images) [ … input vector has a respective dimensionality a range between 100 and 10000] may be used as a training set of the training process. In the training process, the sets of measured signals are successively applied as the input 142 to the machine learning model, and a modification is made to the parameters ϕ and/or θ such that the corresponding output 145 of the auto-encoder is more like this set of measured signals. For example, in the case that the set of measured signals is a diffraction image from one of the locations of the die (e.g. as shown in FIG. 8(a)) [ … input vector has a respective dimensionality a range between 100 and 10000], the number of dimensions in the input vector x may be equal to the number of pixels of the diffraction image. And in [0085] For each die, of each of the seven samples, 100 measurements were made at 100 respective locations on the surface of the die using SXR light. This corresponds to step 601 of method 100. Each location produced a respective pixelated diffraction image, such as the diffraction image shown in FIG. 8(a). FIG. 8(b) is an enlarged view of a portion of the diffraction image of FIG. 8(a). [0086] In step 602, for each die, a respective PCA analysis (which is an example of a component extraction method which is also a dimensionality reduction method) is done of the respective diffraction images for the 100 respective locations on the die, to derive a plurality of PCA components (e.g. five or six components, but it may be higher or lower). The number of PCA components is much lower (e.g. at least 100 times lower, and more normally at least 10000 times lower) than the number of pixels in each of the diffraction images [ … input vector has a respective dimensionality a range between 100 and 10000]…A given diffraction image corresponds to a point in a space having a number of dimensions equal to the number of PCA components, where the coordinates of the point are the respective amplitudes of the PCA components in the image. )
Helf, Hig and Ha are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data for developing methods and systems using machine learning systems to characterize fabrication processes, as disclosed by Helf with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as collectively disclosed by Hig and Ha.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Helf, Hig and Ha as noted above; Doing so allows for developing information processing and modeling techniques to determine parameters of the fabrication process indicative of a fault in the fabrication process, (Helf, Abstract and 0011).
Regarding claim 16, the limitations are similar with claim 6 limitations and are rejected under the same rationales.
Claims 7-9 and 17-18, are rejected under 35 U.S.C. 103 as being unpatentable over Ha et al. (US 20180330511, hereinafter ‘Ha’) in view of Higuchi et al. (US 20220301288, hereinafter ‘Hig’) in view of Kim et al. (US 20230122101 hereinafter ‘Kim’).
Regarding claim 7, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein the latent space is classified into a plurality of clusters using a Gaussian Mixture model. (in [0127] In one such pre-training strategy, as shown in FIG. 7, image 700 generated for a specimen with one modality, e.g., CAD, may be input to encoder 702. Encoder 702 may generate learning based features 704 [wherein the latent space is classified into a plurality of clusters using a ] for image 700. Features 704 are input to decoder 706, which generates reconstructed image 708. Reconstructed image 708 is meant to be the same as the input image. That is, the encoder determines features for image 700, which are then used by the decoder to reconstruct image 700 thereby generating reconstructed image 708, which if the features are determined properly [wherein the latent space is classified into a plurality of clusters using a for determining cluster of appropriate modeled features in the latent space] will be the same as input image 700...)
While Ha teaches processing multi-dimensional input data.
Hig expressly teaches the use of classification models based on the latent space of an autoencoder model to classify the model outputs into a plurality of classification clusters (e.g. classes) as claimed wherein the latent space is classified into a plurality of clusters using a (in [0092] The information processing apparatus 100 inputs each vector included in the vector set 161 to the decoder 153 to generate an image set 162 corresponding to the vector set 161. For example, the image set 162 includes 16 images. The information processing apparatus 100 inputs each image included in the image set 162 to the classification model 141[wherein the latent space is classified into a plurality of clusters using a ] to generate confidence score data 163. The confidence score data 163 includes a row of confidence scores for each image included in the image set 162. Each row of confidence scores lists a plurality of confidence scores corresponding to a plurality of classes [wherein the latent space is classified into a plurality of clusters using a ]. )
While Ha and Hig teach the use of classification models using the latent space of an autoencoder model to classify the model outputs into a plurality of classification clusters (e.g. classes).
Ha and Hig did not expressly disclose the classification model for classifying the latent space used outcome of the autoencoder model as claimed wherein the latent space is classified into a plurality of clusters using a Gaussian Mixture model.
Kim does expressly teach the classification model for classifying the latent space used outcome of the autoencoder model as claimed wherein the latent space is classified into a plurality of clusters using a Gaussian Mixture model. (in As depicted in Fig. 5 And in [0046] The image processing unit 120 may generate a plurality of clusters by applying a clustering algorithm [wherein the latent space is classified into a plurality of clusters using a Gaussian Mixture model.] to the diffraction pattern based on unsupervised learning in operation 220… Unsupervised learning is a method of analyzing or extracting data characteristics of input data without label information, which may utilize an autoencoder (AE) structure [wherein the latent space is classified into a plurality of clusters using a Gaussian Mixture model.].
… [0049] In at least one embodiment, the clustering algorithm may be at least one of a K-means algorithm, a mean shift algorithm, a Gaussian mixture model (GCM) algorithm [wherein the latent space is classified into a plurality of clusters using a Gaussian Mixture model.], a density-based spatial clustering of applications with noise (DBSCAN) algorithm, and/or the like. For example, the image processing unit 120 may apply the K-means algorithm to extract that the optimal number of clusters vector-quantified to the nearest mean in the CBED dataset is, e.g., six. )
Kim, Hig and Ha are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data for developing methods and systems using machine learning systems to help in processing diffraction pattern acquired by scanning an electron beam, as disclosed by Kim with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as collectively disclosed by Hig and Ha.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Kim, Hig and Ha as noted above; Doing so allows for developing information processing and modeling techniques in order to change data to be acquired through the DCNN algorithm and/or to improve (e.g., optimize) the performance and efficiency of the analysis algorithm, (Kim Abstract and 0041).
Regarding claim 8, the rejection of claim 1 is incorporated and Ha in combination with Hig teaches the apparatus of claim 1, wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters(in [0127] In one such pre-training strategy, as shown in FIG. 7, image 700 generated for a specimen with one modality, e.g., CAD, may be input to encoder 702. Encoder 702 may generate learning based features 704 [wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters] for image 700. Features 704 are input to decoder 706, which generates reconstructed image 708. Reconstructed image 708 is meant to be the same as the input image. That is, the encoder determines features for image 700, which are then used by the decoder to reconstruct image 700 thereby generating reconstructed image 708, which if the features are determined properly [wherein the latent space is classified into a plurality of clusters using a for determining cluster of appropriate modeled features in the latent space] will be the same as input image 700...)
Hig expressly teaches use of classification models based on the latent space of an autoencoder model to classify the model outputs into a plurality of classification clusters (e.g. classes) as claimed wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters (in [0092] The information processing apparatus 100 inputs each vector included in the vector set 161 to the decoder 153 to generate an image set 162 corresponding to the vector set 161. For example, the image set 162 includes 16 images. The information processing apparatus 100 inputs each image included in the image set 162 to the classification model 141[wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters] to generate confidence score data 163. The confidence score data 163 includes a row of confidence scores for each image included in the image set 162. Each row of confidence scores lists a plurality of confidence scores corresponding to a plurality of classes [wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters being in a range from four to twenty]. )
While Ha and Hig teach the use of classification models using the latent space of an autoencoder model to classify the model outputs into a plurality of classification clusters (e.g. classes).
Ha and Hig did not expressly disclose the classification model for classifying the latent space used outcome of the autoencoder model as claimed wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters being in a range from four to twenty.
Kim does expressly teach the classification model for classifying the latent space used outcome of the autoencoder model as claimed wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters being in a range from four to twenty. (in As depicted in Fig. 5 and in [0072] For example, the image processing unit 120 may set, as 6, the number of improved or optimal clusters for the plurality of CBED patterns input through the input layer 420a. That is, the image processor 120 may set an arbitrary K value to 6. Afterwards, the image processing unit 120 may arbitrarily select the cluster centroid for the six cluster labels C1, C2, C3, C4, C5, and C6 [wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters being in a range from four to twenty], and allocate each label of the input plurality of CBED patterns as a cluster label corresponding to the nearest centroid. The image processing unit 120 may calculate an average of the CBED patterns belonging to the six clusters to update the six cluster centroids. For example, the image processing unit 120 may repeatedly perform an operation of allocating a plurality of CBED patterns to six clusters and an operation of updating the cluster centroids, thereby generating a cluster map 550 corresponding to the ADF image 400, and divided into six clusters. And in [0049] In at least one embodiment, the clustering algorithm may be at least one of a K-means algorithm, a mean shift algorithm, a Gaussian mixture model (GCM) algorithm, a density-based spatial clustering of applications with noise (DBSCAN) algorithm, and/or the like. For example, the image processing unit 120 may apply the K-means algorithm to extract that the optimal number of clusters vector-quantified to the nearest mean in the CBED dataset is, e.g., six [wherein the latent space is classified into a plurality of clusters, with a number of different clusters in the plurality of clusters being in a range from four to twenty]. )
Kim, Hig and Ha are analogous art because both involve developing information retrieval and data modeling techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for processing data for developing methods and systems using machine learning systems to help in processing diffraction pattern acquired by scanning an electron beam, as disclosed by Kim with the method of developing information retrieval and data modeling techniques using machine learning systems and algorithms as collectively disclosed by Hig and Ha.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Kim, Hig and Ha as noted above; Doing so allows for developing information processing and modeling techniques in order to change data to be acquired through the DCNN algorithm and/or to improve (e.g., optimize) the performance and efficiency of the analysis algorithm, (Kim Abstract and 0041).
Regarding claim 9, the rejection of claim 8 is incorporated and Ha in combination with Hig and Kim teaches the apparatus of claim 8, wherein the electronic controller is configured to generate the cluster-mapped image of the sample by coloring each pixel of the base image in accordance with a color code of the plurality of clusters. (in [0089] Color overlay image 212 may be input to classifier 214, which may be configured to classify the alignment results into either an alignment class or a misalignment class based on color overlay image 212 [wherein the electronic controller is configured to generate the cluster-mapped image of the sample by coloring each pixel of the base image in accordance with a color code of the plurality of clusters]. For example, the classifier may generate a deep learning based alignment metric based on the color overlay image. In another example, GoogLeNet, a very powerful classification architecture known in the art, can be used for the classification task. GoogLeNet may be trained with two classes: Class 1 can mean the cropped images from different modalities are aligned, and Class 2 can mean they are not aligned. A final SoftMax output corresponding to Class 1 may be used as the alignment confidence. For example, a SoftMax of Class 1 (ranged from 0 to 1) is the confidence of the classifier for the alignment results. In this manner, a SoftMax=0 is not confident while a SoftMax=1 is very confident. In addition, to generate data for the misalignment class, random shift errors may be added to the training data for the alignment class. The classifier may perform such classification as described further herein. Classifier 214 may generate output 2 alignment results 216 that include at least the classification of the alignment results. The output of the classifier may be a probability that the two cropped images are aligned (i.e., the confidence metric of the alignment). Each of the elements and steps described and shown in FIG. 2 may be further configured and performed as described further herein.)
Regarding claim 17, the limitations are similar with claim 8 limitations and are rejected under the same rationales.
Regarding claim 18, the limitations are similar with claim 9 limitations and are rejected under the same rationales.
and cause the cluster-mapped image of the sample to be displayed by the display device. (Ha teaches in 0064] As noted above, the optical and electron beam tools may be configured for directing energy (e.g., light, electrons) to and/or scanning energy over a physical version of the specimen thereby generating actual (i.e., not simulated) output and/or images for the physical version of the specimen [cause the cluster-mapped image of the sample to be displayed by the display device]. In this manner, the optical and electron beam tools may be configured as “actual” tools, rather than “virtual” tools. Computer subsystem(s) 102 shown in FIG. 1 may, however, include one or more “virtual” systems 108 that are configured for performing one or more functions using at least some of the actual optical images and/or the actual electron beam images generated for the specimen, which may include any of the one or more functions described further herein. [0065] The one or more virtual systems are not capable of having the specimen disposed therein. In particular, the virtual system(s) are not part of optical tool 10 or electron beam tool 122 and do not have any capability for handling the physical version of the specimen. In other words, in a system configured as a virtual system, the output of its one or more “detectors” may be output that was previously generated by one or more detectors of an actual tool and that is stored in the virtual system, and during the “imaging and/or scanning,” the virtual system may replay the stored output as though the specimen is being imaged and/or scanned [cause the cluster-mapped image of the sample to be displayed by the display device]. In this manner, imaging and/or scanning the specimen with a virtual system may appear to be the same as though a physical specimen is being imaged and/or scanned with an actual system, while, in reality, the “imaging and/or scanning” involves simply replaying output for the specimen in the same manner as the specimen may be imaged and/or scanned.)
Additionally Hig teaches in [0056] The GPU 104 outputs images to a display device 111 connected to the information processing apparatus 100 [cause the cluster-mapped image of the sample to be displayed by the display device] in accordance with commands from the CPU 101. Any kind of display device such as a cathode ray tube (CRT) display, a liquid crystal display (LCD), an organic electro-luminescence (OEL) display, or a projector may be used as the display device 111. Other than the display device 111, an output device such as a printer may be connected to the information processing apparatus 100.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Ha and Hig for the same reasons disclosed above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Loaiza Ganem et al. (US 20230244917): does expressly teach use probability density modeling with autoencoders in data processing task, in [0017] FIG. 1 illustrates a computer modeling system 110 including components for probabilistic modeling of a high-dimensional space, according to one embodiment. The computer modeling system 110 includes computing modules and data stores for generating and using a computer model 160. In particular, the computer model 160 is configured to represent high-dimensional data as a manifold in a low-dimensional space from which a probability density may be learned. That is, the computer model may include an autoencoder model that learns an encoder and decoder for learning a function for transforming a data point in high-dimensional space to and from (the encoder and decoder portions, respectively) a position in low-dimensional space, and a density model for representing the probability density [identify, with the autoencoder, a respective latent-space cluster to which the respective probability density belongs] of the data in the low-dimensional space. By learning the manifold and then the probability density in the low-dimensional space, the probability density may be learned and used to evaluate or produce high-dimensional data without manifold overfitting (e.g., without a training process that learns the manifold but fails to effectively learn the density or vice versa).
Middlebrooks et al. (US 20230004096): teaches a encoder-decoder architecture; wherein the encoder-decoder architecture has an encoding portion (an encoder) and a decoding portion (a decoder).
Pisarenco et al. (US 12586170): teaches inspecting a wafer using a charged particle beam system to generate predictive images of the wafer using machine learning. A CNN encoder is trained by using the mapping to construct the second wafer image from the first wafer image. Trained features of the wafer images may be extracted from the construction and inputted into a CNN decoder.
Phan et al. (US 20230281364): teaches in [0055] FIG. 8 depicts aspects of a temporal-coupling multi-modal mixture model or Mode Fidelity Mixture Model (MFM) in accordance with an embodiment. In FIG. 8, as in FIG. 7, the asset can be operated in different modes, and a data-driven model can use a very few number of sample points to construct a wrong mode. Thus, new conditions are introduced when a mode is established (detected), i.e., to establish a mode, given conditions of “sufficient sample data” and “maximizing loglikelihood”. In FIG. 8, the line 800 denotes a Gaussian mixture model (GMM) which is composed of a weighted sum of three components (Components 1-3) denoted with dashed lines. However, in FIG. 8, these three components correspond to multivariate data 802, 804, 806 composed of the same set of variables during different (possibly at least partially overlapping) windows of time. Thus, the corresponding Gaussian graphical model (GGM) representations 812, 814, 816 of the three components of the GMM (an MFM) each have the same set of nodes (e.g., the set of sensors), with the only difference being the connections between the nodes.
Kaplenko et al. (US 20220037111): teaches methods as defined herein comprises the step of using an alignment algorithm for effecting an alignment transition. Said alignment algorithm may be executed by a processing unit, which may be part of said charged particle beam apparatus, or may be externally connected thereto. The alignment algorithm brings said charged particle beam apparatus from said first alignment state towards a second alignment state. Generally, the second alignment state is an improved alignment state, i.e. a more (optimally) aligned state for carrying out the intended use of said charged particle beam apparatus, although this is not necessarily required. A more unaligned state is conceivable as well. The transition from said first alignment state towards said second alignment state is herein defined as the alignment transition.
Wang et al. (NPL: Image Anomaly Detection Using Normal Data Only by Latent Space Resampling): teaches a novel method only using normal data for image anomaly detection. It effectively excludes the anomalous components in the latent space and avoids the unwanted reconstruction of the anomalous part, which achieves better detection results.
Fukuda et al. (NPL: Anomaly detection in random circuit patterns using autoencoder): teaches an autoencoder is a neural network that has widely been used in various applications including anomaly detections. It consists of an encoder and decoder. The encoder compresses the input data into the middle layer called a latent vector, and the decoder decompresses the data from the latent vector to generate a representation as close to the original input as possible. The laten vector has a smaller dimension than the input data and represents a data-specific and lossy version of the trained data. The network is learned to minimize the reconstruction loss, a distance function, or the amount of information loss between the compressed and the decompressed representations. After training, the autoencoder reconstructs input data only when input data is similar to those used for training data. Thus, by training the autoencoder with normal data, it outputs normal data only when the input is normal, and this characteristic of autoencoders can be used for detecting anomalies.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWATOSIN ALABI whose telephone number is (571)272-0516. The examiner can normally be reached Monday-Friday, 8:00am-5:00pm EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWATOSIN ALABI/Primary Examiner, Art Unit 2129
Prosecution Insights