DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/05/2024 is/are compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Office Action Summary
Claim(s) 1-3, 6-12, and 15-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shkalim et al (US 2022/0254000 A1) in view of Zhang et al (CN 116542984 A; See translation provided by Examiner), further in view of Guo et al (US 2024/0177286 A1).
Claim(s) 4-5, 13-14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shkalim et al (US 2022/0254000 A1) in view of Zhang et al (CN 116542984 A; See translation provided by Examiner) and Guo et al (US 2024/0177286 A1), further in view of Bufi et al (US 2022/0366558 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 6-12, and 15-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shkalim et al (US 2022/0254000 A1) in view of Zhang et al (CN 116542984 A; See translation provided by Examiner), further in view of Guo et al (US 2024/0177286 A1).
Regarding claim(s) 1, 10, and 17, Shkalim teaches a system, comprising:
a computing device (Figure 2; Paragraph [0036]; and Paragraph [0078]); and
a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for automated visual inspection of products for defects (Figure 2; Paragraph [0036]; and Paragraph [0078]), the operations comprising:
receiving a target image depicting a product that is to-be-inspected for defects and a reference image depicting a product that is absent any defects (Paragraph [0037]: “FPEI system 103 is configured to receive, via input interface 105, FP input data […] can include images (e.g. captured images, images derived from the captured images, simulated images, synthetic images, etc.) […]”; and Paragraph [0058]: “[…] MIS processes the acquired aerial images to obtain (603), per defect location (optionally, only for defects in the Regions-of Interest—ROIs), a plurality of defect images and a plurality of reference images related to the respective defect location. The plurality of defect images comprises overlapped patch images of the given defect location. The plurality of reference images comprises patch images of corresponding reference locations in one or more reference dies […]”);
processing the target image through a (Paragraph [0060]: “MIS applies (605) the calibrated PT to the images in the pluralities of defect and reference images, so to generate defect and reference binary print masks informative of printing features on the masks”; and Paragraph [0061]: “MIS can use the binary print masks to provide image segmentation […]”); and
determining that a difference exists between a target mask in the set of target masks and a reference mask in the set of reference masks (Paragraph [0060]: “Subtracting defect and reference print masks results in a print defect map further usable for EPD calculation”; Paragraph [0061]: “the difference between defect and reference binary print masks can be examined only within the ROI”; and Paragraph [0062]: “Defect binary print mask 701 is subtracted from binary reference print mask 702. The resulting edge displacement error 704 is obtained merely within the Region of Interest 703”), and in response:
providing a potential defect patch for a region of interest (ROI) of the target image corresponding to the difference (Paragraph [0058]: “[…] per defect location (optionally, only for defects in the Regions-of Interest—ROIs), a plurality of defect images and a plurality of reference images related to the respective defect location. The plurality of defect images comprises overlapped patch images of the given defect location. The plurality of reference images comprises patch images of corresponding reference locations in one or more reference dies […]”; and Paragraph [0062]: “Defect binary print mask 701 is subtracted from binary reference print mask 702. The resulting edge displacement error 704 is obtained merely within the Region of Interest 703”).
Shkalim fails to teach the segmentation model; generating a potential defect embedding using the potential defect patch, comparing the potential defect embedding to each defect embedding in a set of defect embeddings to provide a set of similarity scores, and selectively indicating presence of a defect in the product based on a similarity score in the set of similarity scores.
However, Zhang teaches the segmentation model (Figure 4: “implementation of the SAM model”; Paragraph [0087]: “the segmentation unit 302 is used to input the image of the hardware to be detected into the segmentation model for image segmentation to obtain a mask area”; and Paragraph [00103]: “Wherein, the segmentation model includes at least one of the vit_h model, the vit_l model, and the vit_b model in the GitHub repository of the SAM model”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine before the effective filing date of the claimed invention to modify the defect inspection method of Shkalim by employing the segmentation model of Zhang to process the defect images and corresponding reference images to generate the respective defect and reference masks. The motivation for this combination of references would have been to improve the accuracy and robustness of generating defect and reference masks for products having different shape, size, and materials. This motivation for the combination of Shkalim and Zhang is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Shkalim and Zhang fails teaches to generating a potential defect embedding using the potential defect patch, comparing the potential defect embedding to each defect embedding in a set of defect embeddings to provide a set of similarity scores, and selectively indicating presence of a defect in the product based on a similarity score in the set of similarity scores.
However, Guo teaches providing a potential defect patch for a region of interest (ROI) of the target image corresponding to the difference (Paragraph [0062]: “The cropping logic 204 performs a cropping procedure with one or more of the received image frames […] The process usually consists of the removal of some of the peripheral areas of an image to remove extraneous trash from the picture, to improve its framing, to change the aspect ratio, or to accentuate or isolate the subject matter from its background”),
generating a potential defect embedding using the potential defect patch (Paragraph [0048]: “the embedding component 194 may receive as input one or more image frames indicative of a substrate processing defect and use historical data 142 with a trained machine learning model 190 to determine a feature embedding corresponding to the image frames”; and Paragraph [0065]: “An embedding is a relatively low-dimensional space into which a high-dimensional representation (e.g., images) can be translated into. The embedding data (e.g., features vectors) capture semantics of the received image frames”),
comparing the potential defect embedding to each defect embedding in a set of defect embeddings to provide a set of similarity scores (Paragraph [0021]: “a plurality of feature vectors representative of previously processed image frames that correspond to various substrate processing defects”; and Paragraph [0066] – Paragraph [0068]: “The searching logic 212 receives embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., image repository). The searching logic 212 performs a proximity search between the received feature embeddings and one or more feature embeddings of the vectorized image […] vector search and/or nearest neighbor solution methodology. The searching tool may identify vectors that are closest to (e.g., most similar to) a received and/or provided feature vector […] the searching tool leverage Euclidean distance and/or Cosine similarity to determine a distance between feature vectors”), and
selectively indicating presence of a defect in the product based on a similarity score in the set of similarity scores (Paragraph [0069]: “The defect detection logic 214 receives one or more similar image frames indicating a substrate processing defect […] identifies a substrate processing defect based on the selection of similar image frames. The defect detection logic 214 identifies an instance of abnormality of a fabrication process based on the comparison between a current image and each of the selection of similar image frames”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine before the effective filing date of the claimed invention to further modify the defect inspection method of Shkalim as modified by Zhang by employing the defect image embedding and similarity search techniques of Guo to characterize and identify a potential defect located within the identified region of interest. The motivation for this combination of references would have been to apply Guo’s embedding-based similarity search to the potential defect patch identified by the combined Shkalim and Zhang system in order to provide faster and more focused defect identification by retrieving previously processed defect images having similar feature representations. This motivation for the combination of Shkalim, Zhange, and Guo is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 2, 11, and 18, Shkalim as modified by Zhange and Guo teaches the method of claim 1, where Guo teaches wherein selectively indicating presence of a defect in the product based on a similarity score in the set of similarity scores comprises:
comparing the similarity score to a threshold similarity score (Paragraph [0105]: “processing logic determines a selection of the plurality of feature vectors based on a proximity between the first feature vector and each of the selection of the plurality of feature vectors”; and Paragraph [0090]: “The image retriever performs a proximity search between the received feature embeddings and one or more feature embeddings […] the image retriever determines a set of proximate images (e.g., within a threshold proximity) using vector search and/or nearest neighbor solution methodology […] the image retriever leverages Euclidean distance and/or Cosine similarity to determine a distance between feature vectors”); and
indicating presence of a defect in response to the similarity score at least meeting the threshold similarity score (Paragraph [0043]: “The defect tool 130 receives one or more similar image frames indicating a substrate processing defect. The defect tool 130 identifies a substrate processing defect based on the selection of similar image frames. The defect tool 130 identifies an instance of abnormality of a fabrication process based on the comparison between a current image and each of the selection of similar image frames”; Paragraph [0105]; and Paragraph [0090]).
Regarding claim(s) 3, 12, and 19, Shkalim as modified by Zhange and Guo teaches the method of claim 2, where Guo teaches wherein the similarity score is a maximum similarity (read as “most similar to”) score in the set of similarity scores (Paragraph [0067]: “the searching logic 212 determines a set of proximate images of an image repository using vector search and/or nearest neighbor solution methodology. The searching tool may identify vectors that are closest to (e.g., most similar to) a received and/or provided feature vector”; Paragraph [0068]: “[…] the searching tool leverage Euclidean distance and/or Cosine similarity to determine a distance between feature vectors”; and Paragraph [0069]: “The defect detection logic 214 receives one or more similar image frames indicating a substrate processing defect. The defect detection logic 214 identifies a substrate processing defect based on the selection of similar image frames”).
Regarding claim(s) 6 and 15, Shkalim as modified by Zhange and Guo teaches the method of claim 1, where Guo teaches wherein generating a potential defect embedding using the potential defect patch comprises processing the potential defect patch through an encoder that embeds the potential defect patch in an embedding space (Figure 6; Paragraph [0094]: “the model architecture 600 is composed of an embedding layer 608, an encoder 614, and a final head classifier 616. Initially, an image is subdivided into non-overlapping patches”; Paragraph [0095]: “at 604 an input image 602 is divided (e.g., flattened) into individual image patches […] The input image may be converted (e.g., flattened) into a sequence 606 of token embedding indicating content of image patches […] the patches are flattened and mapped to n dimensions with a trainable linear projection using linear embedding layer 608”; Paragraph [0096]: “Before feeding the sequence of patches into the encoder 614, it is linearly projected into a vector of the model dimension d using a learned embedding matrix. The embedded representations are then concatenated together […] the positional information 610 is encoded and appended to the patch representations 612 […]”; Paragraph [0097]: “The resulting sequence of embedded patches z.sub.0 is passed to the Transformer encoder 614”; and Paragraph [0098]: “[…] The MSA block 656 determines the relative importance of a single patch embedding with respect to the other embeddings in the sequence”).
Regarding claim(s) 7 and 16, Shkalim as modified by Zhange and Guo teaches the method of claim 6, where Guo teaches wherein each defect embedding in the set of defect embeddings is generated by the encoder (Figure 6; Paragraph [0034]: “Data store 140 may store one or more historical data 142 including housing an image repository 144 of previously processed images (e.g., of substrate defects) and corresponding vectorized image features and metadata 146”; Paragraph [0066]: “The searching logic 212 receives embedding data (e.g., feature vectors, feature embeddings) and determines other feature embeddings (e.g., vectorized image features and metadata) corresponding to previously processed images (e.g., image repository). The searching logic 212 performs a proximity search between the received feature embeddings and one or more feature embeddings of the vectorized image; and Paragraph [0094] – Paragraph [0098])).
Regarding claim(s) 8, Shkalim as modified by Zhange and Guo teaches the method of claim 1, where Zhang teaches wherein the segmentation model comprises a pre-trained, third-party segmentation model (Paragraph [0053]: “the segmentation model includes at least one of the vit_h model, vit_l model, and vit_b model in the GitHub repository of the SAM model”; Paragraph [0056]: “The SAM model can produce high-quality object masks, and prompt types include points, boxes, masks, and text. The SAM model […] can accommodate hardware of different shapes, sizes and materials, and can accurately detect various types of defects, such as cracks, dents and breakages”; and Paragraph [0084]: “the image encoder uses the pre-trained model ViT (VisionTransformer) to process high-resolution input and output a set of image embeddings”).
Regarding claim(s) 9, Shkalim as modified by Zhange and Guo teaches the method of claim 1, wherein determining that a difference exists between a target mask in the set of target masks and a reference mask in the set of reference masks comprises a pixel-wise comparison between pixels of the target mask and pixels of the reference mask (where Shkalim teaches in Paragraph [0060]: “MIS applies (605) the calibrated PT to the images in the pluralities of defect and reference images, so to generate defect and reference binary print masks informative of printing features on the masks. Subtracting defect and reference print masks results in a print defect map further usable for EPD calculation”; Paragraph [0061]: “Thereby, the difference between defect and reference binary print masks can be examined only within the ROI”; and Paragraph [0062]: “Defect binary print mask 701 is subtracted from binary reference print mask 702. The resulting edge displacement error 704 is obtained merely within the Region of Interest 703”; and where Zhang teaches in Paragraph [0052]: “[…] The mask can be viewed as a binary image, where defective areas are marked as foreground, usually represented as white or with a pixel value of 1; non-defective areas are marked as background, usually represented as black or with a pixel value of 0”).
Claim(s) 4-5, 13-14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shkalim et al (US 2022/0254000 A1) in view of Zhang et al (CN 116542984 A; See translation provided by Examiner) and Guo et al (US 2024/0177286 A1), further in view of Bufi et al (US 2022/0366558 A1).
Regarding claim(s) 4, 13, and 20, Shkalim as modified by Zhange and Guo teaches the method of claim 1, but do not specifically teach further comprising providing an output image that depicts the product of the target image with a bounding box indicating a location of the defect in the product and a label indicating a defect type of the defect.
However, Bufi teaches to providing an output image that depicts the product of the target image with a bounding box indicating a location of the defect in the product and a label indicating a defect type of the defect (Paragraph [0317]: “acquired by the imaging unit 122, processed by the node device 148 using object detection techniques to detect and classify defects, and displayed via the user control device 150”; Paragraph [0302]: “Passing the image through the network 156 generates bounding boxes with classes and confidence scores. The bounding box encloses an object (defect) located in the image. The class corresponds to a defect type”; Paragraph [0320]: “[…] The defects are enclosed by bounding boxes 714, 716, and 718 […]”; Paragraph [0322]: “as shown, the user control device 150 may be configured to associate a unique colour indicator with each different defect type. For example, bounding boxes enclosing a particular defect type may be given a particular colour”; and Paragraph [0324]: “[…] the defect data 720 and is configured to generate a visualization displaying the defect data 720”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine before the effective filing date of the claimed invention to further modify the defect inspection method of Shkalim as modified by Zhang and Guo by employing the output visualization technique of Bufi to provide an output image of the inspected product including a bounding box indicating the location of an identified defect and associated defect type information. The motivation for this combination of references would have been to provide a visual inspection output in which each detected defect is spatially localized and associated with corresponding defect type information. This motivation for the combination of Shkalim, Zhang, Guo, and Bufi is/are supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 5 and 14, Shkalim as modified by Zhange, Guo, and Bufi teaches the method of claim 4, wherein the label is determined from a registered defects database and is associated with a defect embedding that resulted in the similarity score (where Guo teaches in Paragraph [0034]: “Data store 140 may store one or more historical data 142 including housing an image repository 144 of previously processed images (e.g., of substrate defects) and corresponding vectorized image features and metadata 146”; Paragraph [0068]: “the searching logic 212 scans a data structure and fetches an image repository and corresponding vectorized image features and metadata […] the searching tool leverage Euclidean distance and/or Cosine similarity to determine a distance between feature vectors”; Paragraph [0093]: “[…] the image retriever 564 identifies one or more other images (e.g., similar image data 566) identified as similar to the input data 562 through the feature data 569”; and Paragraph [0105]: “a proximity between the first feature vector and each of the selection of the plurality of feature vectors”; and where Bufi teaches in Paragraph [0302]: “Passing the image through the network 156 generates bounding boxes with classes and confidence scores. The bounding box encloses an object (defect) located in the image. The class corresponds to a defect type”; and Paragraph [0324]: “[…] the defect data 720 and is configured to generate a visualization displaying the defect data 720”).
Relevant Prior Art Directed to State of Art
Rundo et al (US 2024/0202908 A1) are relevant prior art not applied in the rejection(s) above. Rundo discloses a device, comprising: one or more processors; an unsupervised classification circuitry, configured to execute one or more instructions to: generate, using a non-linear dimensionality reduction machine learning model and based at least in part on vector representations for a set of wafer defect map images, reduced feature data, wherein the reduced feature data comprises feature data for the set of wafer defect map images in a reduced dimensionality space; generate, using a density-based clustering machine learning model and based at least in part on the reduced feature data, one or more wafer defect pattern clusters, wherein each wafer defect map image is associated with a particular wafer defect pattern cluster of the one or more wafer defect pattern clusters; and generate a classification prediction for each wafer defect map image based at least in part on the particular wafer defect pattern cluster associated with the respective wafer defect map image.
Qiu et al (US 2024/0119576 A1) are relevant prior art not applied in the rejection(s) above. Qiu discloses a system, comprising: a memory that stores computer executable components; a processor that executes computer executable components stored in the memory, wherein the computer executable components comprise: an extraction component that extracts multiple layers of features from one or more patches of an image using a pretrained convolutional neural network (CNN); a feature mapping component that concatenates the features from the multiple layers to generate a tensor feature map comprising a one-dimensional feature vector for respective patches; a cropping component that performs center cropping on the tensor feature map; and a calculation component that calculates a distance to a feature distribution mean for respective patches.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONGBONG NAH whose telephone number is (571) 272-1361. The examiner can normally be reached M - F: 9:00 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL MISTRY can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONGBONG NAH/Examiner, Art Unit 2674
Prosecution Insights