DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-17 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 17 is drawn to functional descriptive material recorded on a “machine-readable storage medium”. Normally, the claim would be statutory. However, the broadest reasonable interpretation of a claim drawn to a [Insert the claimed medium - computer-readable medium, computer-readable memory, server, etc.] typically covers forms of non-transitory tangible media as well as transitory propagating signals per se, making the recited claim language directed towards non-statutory subject matter such as a “signal”.
“A transitory, propagating signal … is not a “process, machine, manufacture, or composition of matter.” Those four categories define the explicit scope and reach of subject matter patentable under 35 U.S.C. § 101; thus, such a signal cannot be patentable subject matter.” (In re Nuijten, 84 USPQ2d 1495 (Fed. Cir. 2007)).
Because the full scope of the claim as properly read in light of the disclosure appears to encompass non-statutory subject matter (i.e., because the specification is silent to the exact embodiment of a computer readable medium, it is interpreted as including the ordinary and customary meaning of computer readable medium covering both non-transitory media and transitory propagating signals, etc.) the claim as a whole is non-statutory. In view of the USPTO's Interim Examination Instructions for Evaluating Subject Matter Eligibility under 35 U.S.C. 101 (the "Guidelines"), and the Official Gazette Notice (1351 OG 212, made available February 23, 2010), the examiner suggests amending the claim to include the limitation "non-transitory" in order to exclude any non-statutory subject matter. Any amendment to the claim should be commensurate with its corresponding disclosure.
35 U.S.C. § 112 Sixth Paragraph - Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “module” and “sub-module” in claims 13-14.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 8 and 13 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Huang et al. (US PGPub 2020/0104650, hereby referred to as “Huang”).
Consider Claims 1, 8 and 13.
Huang teaches:
1. A method for automatic optical inspection (AOI), comprising: / 8. A method of training an inspection model for automatic optical inspection (AOI), comprising: / 13. A device for automatic optical inspection (AOI), comprising: (Huang: abstract, A fusion-based classifier, classification method, and classification system, wherein the classification method includes: generating a plurality of probability vectors according to input data, wherein each of the plurality of probability vectors includes a plurality of elements corresponding to a plurality of class respectively; selecting, from the plurality of probability vectors, a first probability vector having an extremum value corresponding to a first class-of-interest according to the first class-of-interest; and determining a class of the input data according to the first probability vector.[0007] The disclosure provides a classification system based on probability fusion, including: an automatic optical detection device and a processor. The automatic optical inspection device obtains image data of an article. The processor is configured to control a classifier, and the classifier includes: a sub-classifier, a fusion layer, and an output layer. The sub-classifier generates a plurality of probability vectors according to the image data, wherein each of the plurality of probability vectors includes a plurality of elements corresponding to a plurality of classes. The fusion layer selects a first probability vector having an extremum value corresponding to a first class-of-interest from the plurality of probability vectors according to the first class-of-interest. The output layer determines a class of an appearance defect according to the first probability vector.[0030]-[0047], [0030] FIGS. 1 A is schematic diagrams of a fusion-based classifier 150 according to an embodiment of the disclosure, wherein the classifier 150 is adapted to classify input data into one of a plurality of classes, and the classifier 150 may be implemented by a hardware (e.g.: a circuit or an integrated circuit) or a software (e.g.: one or more modules stored in a storage medium), and the disclosure is not limited thereto.)
1. obtaining an image of an inspected component of a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; / 8. collecting images of industrial components in a plurality of domains as a multi-domain AOI training dataset, wherein the plurality of domains correspond to a plurality of industrial component types, respectively or correspond to a plurality of industrial component production lines, respectively; / 13. an image obtaining module configured to obtain an image of an inspected component of a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; (Huang: [0033] The input data IN may be any type of data. For example, the input data IN may include a feature map output from a convolution neural network (CNN). In some embodiments, the input data IN may also be output data of other kinds of neural networks, which may include an autoencoder neural network, a deep learning neural network, and a deep residual learning neural network, Boltzmann machine (RBM) neural network, recursive neural network or multi-layer perceptron (MLP) neural network, etc., the disclosure is not limited thereto. If the classifier 150 is used in the manufacturing of wafer fabrication, semiconductor manufacturing or printed circuit board (PCB), the input data IN may be, for example, an image data of an appearance of a wafer to be inspected obtained automatic optical inspection equipment (AOI), or an image data of an appearance of a printed circuit board obtained by automatic visual inspection (AVI) equipment, the disclosure is not limited thereto. [0030]-[0047], Figure 1A [0039] In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market.)
1. inputting the image of the inspected component into an inspection model, the inspection model comprising a generic feature extraction sub-model and at least one task inspection sub-model associated with a particular AOI task of the first domain; / 8. training the generic feature extraction sub-model in the inspection model based on the multi-domain AOI training dataset to obtain a trained generic feature extraction sub-model; / 13. an inspection module, the inspection module comprising a generic feature extraction sub-module and at least one task inspection sub-module associated with a particular AOI task of the first domain, (Huang: [0030]-[0047], Figures 1A; [0031] The classifier 150 may include a sub-classifier 151, a fusion layer 152, and an output layer 153. In this embodiment, it is assumed that input data IN includes n pieces of data such as data i1, data i2, . . . , and data in. The sub-classifier 151 generates a plurality of probability vectors v 1, v2, . . . , and vn according to the input data IN. Specifically, the sub-classifier 151 may receive the data i1 and classify the data i1 to generate a score vector corresponding to the data i1. The size of the score vector depends on the number of classes that the classifier 150 may discern. Assuming the classifier 150 may classify the input data IN into one of M classes, the size of the score vector is M×1, wherein each element in the score vector represents a score value of a certain class corresponding to the data i1. [0032] After calculating the score vector of the data i1, the sub-classifier 151 may convert the score vector into a probability vector through a softmax function. The probability vector also has M elements. Each of the M elements represents a probability value corresponding to a class of the data i1, and the probability value is between 0 and 1. After repeating the above steps, the sub-classifier 151 may generate the probability vectors v1, v2, . . . , and vn based on the data i1, the data i2, . . . , and the data in, respectively. [0034] Although in the embodiment of FIG. 1A, a plurality of probability vectors v1, v2, . . . , and vn are generated by a single sub-classifier 151, the disclosure is not limited thereto. For example, the plurality of probability vectors v1, v2, . . . , and vn may also be generated by different sub-classifiers 1511, 1512, . . . , and 151 n according to data i1, i2, . . . , and in in input data IN, respectively, as shown in FIG. 1B. [0035] Returning to FIG. 1A, the fusion layer 152 may select a first probability vector p1 having an extremum value corresponding to a first class-of-interest from the plurality of probability vectors according to the first class-of-interest, wherein the first class-of-interest corresponds to one of a plurality of elements in each of the plurality of probability vectors.)
1. generating, by the generic feature extraction sub-model, a first feature representation of the image based on the image of the inspected component; / 8. training at least one task inspection sub-model associated with a particular AOI task of a first domain in the inspection model based on an image of an industrial component in the first domain as a single-domain AOI training dataset, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line, / 13. the generic feature extraction sub-module configured to generate a first feature representation of the image based on the image of the inspected component, (Huang: [0034]-[0035] Returning to FIG. 1A, the fusion layer 152 may select a first probability vector p1 having an extremum value corresponding to a first class-of-interest from the plurality of probability vectors according to the first class-of-interest, wherein the first class-of-interest corresponds to one of a plurality of elements in each of the plurality of probability vectors. [0036] Specifically, after the plurality of probability vectors v1, v2, . . . , and vn are generated in the sub-classifier 151, the fusion layer 152 may obtain an element corresponding to the first class-of-interest in each probability vector (it is assumed below that each probability vector has M elements, and each of the M elements corresponds to a different class, where element k corresponds to first class-of-interest). Next, the fusion layer 152 may select an element k having an extremum value from among n element k corresponding to n probability vectors (i.e., probability vector v1, v2, . . . , and vn), where k is one of 1 to n. The probability vector corresponding to the element k with the extremum value may be used as first probability vector p1. In the disclosure, the extremum value may represent a maximum value and a minimum value. For example, suppose the classifier 150 in FIG. 1A classifies the input data IN into one of five classes (i.e.: M=5), and the input data IN includes five pieces of data (i.e.: n=5). For the data i1, i2, i3, i4, and i5, the sub-classifier 151 may generate 5 probability vector v1, v2, v3, v4, and v5 according to the data, and the 5 probability vectors may be expressed as an example of a probability matrix of equation (1), as follows:
PNG
media_image1.png
116
1055
media_image1.png
Greyscale
[0037] In the equation (1), the matrix to the right of the equal sign is called the probability matrix. The element in the probability matrix is represented as where Vx,y, represents a row number in which an element V is located and an index of a class corresponding to the element V. y represents a column number in which element V is located and an index of a probability vector corresponding to the element V. For example, V3,1=0.569379 represents a probability value of element (i.e.: V3,1) corresponding to class 3 in the probability vector v1 is 0.569379. For another example, V2,4=0.000003 represents a probability value of element (i.e.: V2,4) corresponding to class 2 in the probability vector v4 is 0.000003. The representation of the remaining elements may be deduced by analogy, and will not be repeated here. In this embodiment, it is assumed that the user is interested in a class with an index of 5 in M (M=5) classes (hereinafter referred to as class 5), and the user may set class 5 as first class-of-interest (i.e.: element k=5).)
1. and generating, by the at least one task inspection sub-model, at least one inspection result associated with the particular AOI task of the first domain based on the first feature representation of the image./ 8. wherein the trained generic feature extraction sub-model receives an image in the single-domain AOI training dataset and outputs a first feature representation of the image, and the at least one task inspection sub-model receives the first feature representation of the image and predicts at least one inspection result associated with the particular AOI task of the first domain./ 13. and the at least one task inspection sub-module configured to generate at least one inspection result associated with the particular AOI task of the first domain based on the first feature representation of the image. (Huang: [0030]-[0047], Figures 1A; [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0040] Take the equation (1) as an example. In order to improve the precision rate of the first class-of-interest (i.e.: class 5), the fusion layer 152 may select V5,2 having a minimum value from the elements V5,1, V5,2, V5,3, V5,4 and V5,5 , according to the first class-of-interest (i.e.: element k=5), and the probability vector v2 corresponding to V5,2 is taken as the as the first probability vector p1. The method of converting the plurality of probability vectors (i.e., probability vector v1, v2, v3, v4, and v5) into a first probability vector pl by using the minimum value of the element of the plurality of probability vectors is referred to herein as class-of-interest minimum fusion (COIMin-Fusion). In this context, COIMax-Fusion and COIMin-Fusion may be collectively referred to as class-of-interest maximum/minimum fusion (COIM-Fusion).)
Consider Claim 2.
Huang teaches: 2. The method of claim 1, wherein the particular AOI task of the first domain comprises at least one or at least two of the following tasks: a classification task for classifying the image of the inspected component into one class of a plurality of classes; a segmentation task for segmenting a defective area in the image of the inspected component; and an identification task for identifying the defective area in the image of the inspected component. (Huang: [0039]-[0041], [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0041] After obtaining the first probability vector p1 (i.e., probability vector v2), the output layer 153 may receive the first probability vector p1, and determine the class of the input data IN according to the first probability vector p 1. In the embodiment, among the elements (V1,2, V2,2, V3,2, V4,2 and V5,2) of the first probability vector p1 (i.e., probability vector v2), the class 1 has the largest probability value (V1,2=0.781814). Based on this, the output layer 153 may classify the input data IN into class 1. Increasing the precision rate of a defect class may effectively reduce the false discovery rate (FDR) of the defect class. In this way, the number of samples of the product having the defect class may be reduced, thereby reducing the burden on the quality review personnel to perform manual re-examination of the sample. [0048] FIG. 2 is a schematic diagram of a classification system 10 based on probability fusion according to an embodiment of the disclosure. Classification system 10 is adapted to classifying the appearance defect of an item (e.g.: wafer) or the appearance defect of a printed circuit board into one of a variety of classes. The classification system 10 may include an automatic optical inspection equipment (or automatic visual inspection equipment) 110, a processor 130, and a classifier 150 (as shown in FIG. 1A). Automatic optical inspection (AOI) is a high-speed, high-accuracy optical image detection system that uses “mechanical vision” as an object to detect and replace human eye, brain or hand movements to detect the quality of the product or whether there are defects, etc. Automatic optical inspection technology is a non-contact detection technology that uses automatic optical inspection equipment to obtain the surface state of semi-finished or finished products (e.g.: wafers), and then uses image processing technology to detect defects such as foreign objects or pattern anomalies. Automatic optical inspection technology may improve the traditional drawbacks of using optical instruments for human detection. The automatic optical inspection equipment 110 is used to obtain image data of the appearance of a wafer or a printed circuit board. The image data may be used as the input data IN of the classifier 150.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-17 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. (US PGPub 2020/0104650, hereby referred to as “Huang”, in view of Sathyendra et al. (US PGPub 20150347871, hereby referred to as “Sathyendra”.
Consider Claims 1, 8 and 13.
Huang teaches:
1. A method for automatic optical inspection (AOI), comprising: / 8. A method of training an inspection model for automatic optical inspection (AOI), comprising: / 13. A device for automatic optical inspection (AOI), comprising: (Huang: abstract, A fusion-based classifier, classification method, and classification system, wherein the classification method includes: generating a plurality of probability vectors according to input data, wherein each of the plurality of probability vectors includes a plurality of elements corresponding to a plurality of class respectively; selecting, from the plurality of probability vectors, a first probability vector having an extremum value corresponding to a first class-of-interest according to the first class-of-interest; and determining a class of the input data according to the first probability vector.[0007] The disclosure provides a classification system based on probability fusion, including: an automatic optical detection device and a processor. The automatic optical inspection device obtains image data of an article. The processor is configured to control a classifier, and the classifier includes: a sub-classifier, a fusion layer, and an output layer. The sub-classifier generates a plurality of probability vectors according to the image data, wherein each of the plurality of probability vectors includes a plurality of elements corresponding to a plurality of classes. The fusion layer selects a first probability vector having an extremum value corresponding to a first class-of-interest from the plurality of probability vectors according to the first class-of-interest. The output layer determines a class of an appearance defect according to the first probability vector.[0030]-[0047], [0030] FIGS. 1 A is schematic diagrams of a fusion-based classifier 150 according to an embodiment of the disclosure, wherein the classifier 150 is adapted to classify input data into one of a plurality of classes, and the classifier 150 may be implemented by a hardware (e.g.: a circuit or an integrated circuit) or a software (e.g.: one or more modules stored in a storage medium), and the disclosure is not limited thereto.)
1. obtaining an image of an inspected component of a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; / 8. collecting images of industrial components in a plurality of domains as a multi-domain AOI training dataset, wherein the plurality of domains correspond to a plurality of industrial component types, respectively or correspond to a plurality of industrial component production lines, respectively; / 13. an image obtaining module configured to obtain an image of an inspected component of a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; (Huang: [0033] The input data IN may be any type of data. For example, the input data IN may include a feature map output from a convolution neural network (CNN). In some embodiments, the input data IN may also be output data of other kinds of neural networks, which may include an autoencoder neural network, a deep learning neural network, and a deep residual learning neural network, Boltzmann machine (RBM) neural network, recursive neural network or multi-layer perceptron (MLP) neural network, etc., the disclosure is not limited thereto. If the classifier 150 is used in the manufacturing of wafer fabrication, semiconductor manufacturing or printed circuit board (PCB), the input data IN may be, for example, an image data of an appearance of a wafer to be inspected obtained automatic optical inspection equipment (AOI), or an image data of an appearance of a printed circuit board obtained by automatic visual inspection (AVI) equipment, the disclosure is not limited thereto. [0030]-[0047], Figure 1A [0039] In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market.)
1. inputting the image of the inspected component into an inspection model, the inspection model comprising a generic feature extraction sub-model and at least one task inspection sub-model associated with a particular AOI task of the first domain; / 8. training the generic feature extraction sub-model in the inspection model based on the multi-domain AOI training dataset to obtain a trained generic feature extraction sub-model; / 13. an inspection module, the inspection module comprising a generic feature extraction sub-module and at least one task inspection sub-module associated with a particular AOI task of the first domain, (Huang: [0030]-[0047], Figures 1A; [0031] The classifier 150 may include a sub-classifier 151, a fusion layer 152, and an output layer 153. In this embodiment, it is assumed that input data IN includes n pieces of data such as data i1, data i2, . . . , and data in. The sub-classifier 151 generates a plurality of probability vectors v 1, v2, . . . , and vn according to the input data IN. Specifically, the sub-classifier 151 may receive the data i1 and classify the data i1 to generate a score vector corresponding to the data i1. The size of the score vector depends on the number of classes that the classifier 150 may discern. Assuming the classifier 150 may classify the input data IN into one of M classes, the size of the score vector is M×1, wherein each element in the score vector represents a score value of a certain class corresponding to the data i1. [0032] After calculating the score vector of the data i1, the sub-classifier 151 may convert the score vector into a probability vector through a softmax function. The probability vector also has M elements. Each of the M elements represents a probability value corresponding to a class of the data i1, and the probability value is between 0 and 1. After repeating the above steps, the sub-classifier 151 may generate the probability vectors v1, v2, . . . , and vn based on the data i1, the data i2, . . . , and the data in, respectively. [0034] Although in the embodiment of FIG. 1A, a plurality of probability vectors v1, v2, . . . , and vn are generated by a single sub-classifier 151, the disclosure is not limited thereto. For example, the plurality of probability vectors v1, v2, . . . , and vn may also be generated by different sub-classifiers 1511, 1512, . . . , and 151 n according to data i1, i2, . . . , and in in input data IN, respectively, as shown in FIG. 1B. [0035] Returning to FIG. 1A, the fusion layer 152 may select a first probability vector p1 having an extremum value corresponding to a first class-of-interest from the plurality of probability vectors according to the first class-of-interest, wherein the first class-of-interest corresponds to one of a plurality of elements in each of the plurality of probability vectors.)
1. generating, by the generic feature extraction sub-model, a first feature representation of the image based on the image of the inspected component; / 8. training at least one task inspection sub-model associated with a particular AOI task of a first domain in the inspection model based on an image of an industrial component in the first domain as a single-domain AOI training dataset, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line, / 13. the generic feature extraction sub-module configured to generate a first feature representation of the image based on the image of the inspected component, (Huang: [0034]-[0035] Returning to FIG. 1A, the fusion layer 152 may select a first probability vector p1 having an extremum value corresponding to a first class-of-interest from the plurality of probability vectors according to the first class-of-interest, wherein the first class-of-interest corresponds to one of a plurality of elements in each of the plurality of probability vectors. [0036] Specifically, after the plurality of probability vectors v1, v2, . . . , and vn are generated in the sub-classifier 151, the fusion layer 152 may obtain an element corresponding to the first class-of-interest in each probability vector (it is assumed below that each probability vector has M elements, and each of the M elements corresponds to a different class, where element k corresponds to first class-of-interest). Next, the fusion layer 152 may select an element k having an extremum value from among n element k corresponding to n probability vectors (i.e., probability vector v1, v2, . . . , and vn), where k is one of 1 to n. The probability vector corresponding to the element k with the extremum value may be used as first probability vector p1. In the disclosure, the extremum value may represent a maximum value and a minimum value. For example, suppose the classifier 150 in FIG. 1A classifies the input data IN into one of five classes (i.e.: M=5), and the input data IN includes five pieces of data (i.e.: n=5). For the data i1, i2, i3, i4, and i5, the sub-classifier 151 may generate 5 probability vector v1, v2, v3, v4, and v5 according to the data, and the 5 probability vectors may be expressed as an example of a probability matrix of equation (1), as follows:
PNG
media_image1.png
116
1055
media_image1.png
Greyscale
[0037] In the equation (1), the matrix to the right of the equal sign is called the probability matrix. The element in the probability matrix is represented as where Vx,y, represents a row number in which an element V is located and an index of a class corresponding to the element V. y represents a column number in which element V is located and an index of a probability vector corresponding to the element V. For example, V3,1=0.569379 represents a probability value of element (i.e.: V3,1) corresponding to class 3 in the probability vector v1 is 0.569379. For another example, V2,4=0.000003 represents a probability value of element (i.e.: V2,4) corresponding to class 2 in the probability vector v4 is 0.000003. The representation of the remaining elements may be deduced by analogy, and will not be repeated here. In this embodiment, it is assumed that the user is interested in a class with an index of 5 in M (M=5) classes (hereinafter referred to as class 5), and the user may set class 5 as first class-of-interest (i.e.: element k=5).)
1. and generating, by the at least one task inspection sub-model, at least one inspection result associated with the particular AOI task of the first domain based on the first feature representation of the image./ 8. wherein the trained generic feature extraction sub-model receives an image in the single-domain AOI training dataset and outputs a first feature representation of the image, and the at least one task inspection sub-model receives the first feature representation of the image and predicts at least one inspection result associated with the particular AOI task of the first domain./ 13. and the at least one task inspection sub-module configured to generate at least one inspection result associated with the particular AOI task of the first domain based on the first feature representation of the image. (Huang: [0030]-[0047], Figures 1A; [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0040] Take the equation (1) as an example. In order to improve the precision rate of the first class-of-interest (i.e.: class 5), the fusion layer 152 may select V5,2 having a minimum value from the elements V5,1, V5,2, V5,3, V5,4 and V5,5 , according to the first class-of-interest (i.e.: element k=5), and the probability vector v2 corresponding to V5,2 is taken as the as the first probability vector p1. The method of converting the plurality of probability vectors (i.e., probability vector v1, v2, v3, v4, and v5) into a first probability vector pl by using the minimum value of the element of the plurality of probability vectors is referred to herein as class-of-interest minimum fusion (COIMin-Fusion). In this context, COIMax-Fusion and COIMin-Fusion may be collectively referred to as class-of-interest maximum/minimum fusion (COIM-Fusion).)
2. The method of claim 1, wherein the particular AOI task of the first domain comprises at least one or at least two of the following tasks: a classification task for classifying the image of the inspected component into one class of a plurality of classes; a segmentation task for segmenting a defective area in the image of the inspected component; and an identification task for identifying the defective area in the image of the inspected component. (Huang: [0039]-[0041], [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0041] After obtaining the first probability vector p1 (i.e., probability vector v2), the output layer 153 may receive the first probability vector p1, and determine the class of the input data IN according to the first probability vector p 1. In the embodiment, among the elements (V1,2, V2,2, V3,2, V4,2 and V5,2) of the first probability vector p1 (i.e., probability vector v2), the class 1 has the largest probability value (V1,2=0.781814). Based on this, the output layer 153 may classify the input data IN into class 1. Increasing the precision rate of a defect class may effectively reduce the false discovery rate (FDR) of the defect class. In this way, the number of samples of the product having the defect class may be reduced, thereby reducing the burden on the quality review personnel to perform manual re-examination of the sample. [0048] FIG. 2 is a schematic diagram of a classification system 10 based on probability fusion according to an embodiment of the disclosure. Classification system 10 is adapted to classifying the appearance defect of an item (e.g.: wafer) or the appearance defect of a printed circuit board into one of a variety of classes. The classification system 10 may include an automatic optical inspection equipment (or automatic visual inspection equipment) 110, a processor 130, and a classifier 150 (as shown in FIG. 1A). Automatic optical inspection (AOI) is a high-speed, high-accuracy optical image detection system that uses “mechanical vision” as an object to detect and replace human eye, brain or hand movements to detect the quality of the product or whether there are defects, etc. Automatic optical inspection technology is a non-contact detection technology that uses automatic optical inspection equipment to obtain the surface state of semi-finished or finished products (e.g.: wafers), and then uses image processing technology to detect defects such as foreign objects or pattern anomalies. Automatic optical inspection technology may improve the traditional drawbacks of using optical instruments for human detection. The automatic optical inspection equipment 110 is used to obtain image data of the appearance of a wafer or a printed circuit board. The image data may be used as the input data IN of the classifier 150.)
Huang doesn’t teach: limitations from claim 3:
3. The method of claim 2, wherein when the particular AOI task of the first domain is the classification task, the task inspection sub-model is a multi-layer perceivorator (MLP) model; when the particular AOI task of the first domain is the segmentation task, the task inspection sub-model is a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism; and when the particular AOI task of the first domain is the identification task, the task inspection sub-model is a boundary frame decoder with a classification result of the CNN model or the decoder based on the attention mechanism.
Satheyendra teaches:
1. A method for automatic optical inspection (AOI), comprising: / 8. A method of training an inspection model for automatic optical inspection (AOI), comprising: / 13. A device for automatic optical inspection (AOI), comprising: (Satheyendra: abstract, A system and method for performing Automatic Target Recognition by combining the outputs of several classifiers. In one embodiment, feature vectors are extracted from radar images and fed to three classifiers. The classifiers include a Gaussian mixture model neural network, a radial basis function neural network, and a vector quantization classifier. The class designations generated by the classifiers are combined in a weighted voting system, i.e., the mode of the weighted classification decisions is selected as the overall class designation of the target. A confidence metric may be formed from the extent to which the class designations of the several classifiers are the same. This system is also designed to handle unknown target types and subsequent re-integration at a later time, effectively, artificially and automatically increasing the training database size.)
1. obtaining an image of an inspected component of a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; / 8. collecting images of industrial components in a plurality of domains as a multi-domain AOI training dataset, wherein the plurality of domains correspond to a plurality of industrial component types, respectively or correspond to a plurality of industrial component production lines, respectively; / 13. an image obtaining module configured to obtain an image of an inspected component of a first domain, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line; (Satheyendra: [0004] Inverse synthetic aperture radar (ISAR) is a signal processing technique used to form a two-dimensional (2-D) radar image from moving target objects by separating radar returns in Doppler frequency and in range. ISAR is possible with or without radar platform motion. An ISAR 2-D image is comprised of different intensity pixels of reflected point scatterers located at particular range and Doppler bin indices. [0021] According to an embodiment of the present invention there is provided a system for automatic target recognition of a target, the system including a processing unit configured to: receive a sequence of imaging radar images of the target; form a feature vector including measured characteristics of the target; perform a first target recognition attempt, the performing of the first target recognition attempt including: using a Gaussian mixture model neural network classifier to generate a first plurality of probability likelihoods, each of the first plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a first set of class designation rules to produce a first class designation, the first class designation corresponding to one of the plurality of candidate target types; perform a second target recognition attempt, the performing of the second target recognition attempt including: using a radial basis function neural network classifier to generate a second plurality of probability likelihoods, each of the second plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a second set of class designation rules to produce a second class designation, the second class designation corresponding to one of the plurality of candidate target types; perform a third target recognition attempt, the performing of the third target recognition attempt including: using a vector quantization classifier to generate a third plurality of probability likelihoods, each of the third plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a third set of class designation rules to produce a third class designation, the third class designation corresponding to one of the plurality of candidate target types; and combine the first class designation, the second class designation and the third class designation to generate an overall class designation. [0036] FIG. 1 represents an overall block diagram for an ATR engine process according to one embodiment. Data inputs include radar data 110 and the trained classifier parameters 115, which are fed into the signal processing block 120 to attain the target designations for the radar data 110. Off-line training is used to determine the trained classifier parameters 115 that can then be used in real time to determine the class designation for a given input image. The signal processing block includes an image formation block 125 that creates the input 2-D image. The second step is to determine, in a good frame selection block 130, the suitability of the input image for the classification problem. If the image is deemed suitable, a feature vector is extracted from the image in a multi-frames features block 135. A stored history of one-dimensional (1-D) features from previous frames also aids in this feature vector extraction process. The extracted features together act as inputs for the predictive classification mapper 140 that determines a target class.)
1. inputting the image of the inspected component into an inspection model, the inspection model comprising a generic feature extraction sub-model and at least one task inspection sub-model associated with a particular AOI task of the first domain; / 8. training the generic feature extraction sub-model in the inspection model based on the multi-domain AOI training dataset to obtain a trained generic feature extraction sub-model; / 13. an inspection module, the inspection module comprising a generic feature extraction sub-module and at least one task inspection sub-module associated with a particular AOI task of the first domain, (Satheyendra: [0034] In one embodiment, an ATR problem starts by identifying relevant target features that can be extracted. For an ISAR image this is particularly important, because of the ambiguities in the Doppler dimension. The range dimension more aptly represents physical distances of targets, and thus can be used more directly for feature extraction purposes. Apparent length (La) is the length as determined from a target in an ISAR image. A corresponding true physical length is found by dividing the apparent length by the cosine of its aspect angle Θasp, which is defined as the angle formed by the radar line of sight (LOS) and the longitudinal axis of the target. This angle is a function of azimuth as well as elevation. Aside from length, a maritime target can have other distinguishing features, such as mast(s), superstructure(s), rotating object(s), reflector(s), etc. These locations, which may be referred to as points of interest (POIs) can be used in distinguishing targets and discriminating between them.
[0035] Feature extraction is important when distinguishing between classes. In one embodiment, the process begins by isolating the target region using a segmentation technique; edges are then detected using target lines found by Hough processing. [0036] FIG. 1 represents an overall block diagram for an ATR engine process according to one embodiment. Data inputs include radar data 110 and the trained classifier parameters 115, which are fed into the signal processing block 120 to attain the target designations for the radar data 110. Off-line training is used to determine the trained classifier parameters 115 that can then be used in real time to determine the class designation for a given input image. The signal processing block includes an image formation block 125 that creates the input 2-D image. The second step is to determine, in a good frame selection block 130, the suitability of the input image for the classification problem. If the image is deemed suitable, a feature vector is extracted from the image in a multi-frames features block 135. A stored history of one-dimensional (1-D) features from previous frames also aids in this feature vector extraction process. The extracted features together act as inputs for the predictive classification mapper 140 that determines a target class. [0037] The inputs to the image formation block 125 are radar video phase histories and auxiliary data. In this embodiment ISAR is used to form a 2-D image with Doppler on the y-axis and range on the x-axis. Good frame selection pre-screening is crucial for correct classification. The first image screening approach makes sure that the target aspect angle Θasp occurs in the interval of −45 ≦Θasp ≦45 degrees. If this is not the case then the image is rejected as an unsuitable frame for classification. Target aspect angles outside of this bound lead to the target being warped in either the range dimension or the Doppler dimension, or both, which results in erroneous determinations of true length and similar parameters. [0038] FIG. 2 represents the top level block diagram for the frame's feature extraction algorithms, wherein the dashed area, representing the multi-frames features block 135, repeats for each frame.)
1. generating, by the generic feature extraction sub-model, a first feature representation of the image based on the image of the inspected component; / 8. training at least one task inspection sub-model associated with a particular AOI task of a first domain in the inspection model based on an image of an industrial component in the first domain as a single-domain AOI training dataset, the first domain corresponding to a first industrial component type or corresponding to a first industrial component production line, / 13. the generic feature extraction sub-module configured to generate a first feature representation of the image based on the image of the inspected component, (Satheyendra: [0038] FIG. 2 represents the top level block diagram for the frame's feature extraction algorithms, wherein the dashed area, representing the multi-frames features block 135, repeats for each frame. The isolate target region block 210 isolates a rough silhouette of the target region, which acts as a submask in conjunction with the input image to form the input to the length estimation and Hough processing block 215. The target length is estimated and the target range region is further refined, and acts as input to the Hough processing algorithms. The Hough peaks and Hough lines for the targets are extracted, as well as a refined target length estimate, which act as input to the feature vector generation block 220. The feature vector generation block 220 constructs the frame's feature vector, which is used for classifier training and for testing. Training occurs off line and with the multi-frames features block 135 processing repeating for each training frame. Training may be performed with real radar data, e.g., ISAR data obtained in the field, to avoid the disadvantages associated with using simulated data for training. A set of real data may then be separated into a subset used for training, and a second subset used for testing, to assess the performance of the ATR system after training. Testing occurs in real time and for a single frame, which is processed through the full ATR engine, until a class designation is determined.)
1. and generating, by the at least one task inspection sub-model, at least one inspection result associated with the particular AOI task of the first domain based on the first feature representation of the image./ 8. wherein the trained generic feature extraction sub-model receives an image in the single-domain AOI training dataset and outputs a first feature representation of the image, and the at least one task inspection sub-model receives the first feature representation of the image and predicts at least one inspection result associated with the particular AOI task of the first domain./ 13. and the at least one task inspection sub-module configured to generate at least one inspection result associated with the particular AOI task of the first domain based on the first feature representation of the image. (Satheyendra: [0039]-[0044], Figures 3-4, [0041] FIG. 4 shows a block diagram for a predictive classification mapper employing a Gaussian mixture model neural network (GMM-NN) method. The parameters that need to be defined for the GMM-NN are the number of mixture components or centers in the model, the type of model, such as spherical, as well as its dimensionality. The Netlab neural network toolbox, part of the MATLAB™ package, available from The Mathworks, of Natick, Mass., may be used for the creation of the GMM with the ‘gmm’ function. GMM initialization then occurs by using the ‘gmminit’ function in the same toolbox. The expectation maximization (EM) algorithm is used to train the GMM and represent the class's input data as a mixture of weighted Gaussian components. [0042] For testing purposes, each class 410 in the database (6 in one embodiment), as well as a class of unknown targets of significantly larger size (referred to herein as an “unknown large” target type) and a class of unknown targets of significantly smaller size (referred to herein as an “unknown small” target type) are represented by a set of GMM parameters, generated using the expectation maximization algorithm. )
2. The method of claim 1, wherein the particular AOI task of the first domain comprises at least one or at least two of the following tasks: a classification task for classifying the image of the inspected component into one class of a plurality of classes; a segmentation task for segmenting a defective area in the image of the inspected component; and an identification task for identifying the defective area in the image of the inspected component. (Satheyendra: [0035] Feature extraction is important when distinguishing between classes. In one embodiment, the process begins by isolating the target region using a segmentation technique; edges are then detected using target lines found by Hough processing. [0039]-[0044], Figures 3-4, [0039] FIG. 3 represents the block diagram, in one embodiment, for the creation of a frame's feature vector. The Hough lines are initially used to isolate the target's center line in the image, its image line, and the refined length. The refined length estimate is formed by processing of the input sub-masked image using sum normalized range profile (SNRP) methodology. This length estimate is averaged (via median operator) with the previous 4 length determinations and is the first target feature. The purpose of the median operator is to avoid instantaneous erroneous fluctuations in the length estimates such as land returns and/or noise spikes. Using the locations of Hough lines and respective (line) designations, pertinent vertical lines are extracted. These lines are then compared with the locations of the peaks associated with the stored SNRP summed profile. If they occur in the same region, then the lines are designated as possible point of interest (POI) locations. If the Doppler extent is greater than a pre-determined threshold, the POI is deemed to be a possible reflector/rotator; otherwise it is deemed to be a possible mast/superstructure. Additional POI weighting occurs if high correlation exists with the SNRP peak locations. One range centroid of the POIs is stored for every target region; each such range centroid is then designated with a weight. If no POI exists in a region, the weighting is zero. The type designations for the POI are 0 for no POI, 1 for superstructure/mast, or 2 for rotator/reflector. The use of multiple regions may be advantageous in converting an otherwise 2-D feature extraction process into a 1-D process, in which the use of standard and more intricate classifiers can occur. This projection also speeds up the classification process. For larger targets, the use of more sections can also lead to a more refined POI range extent determination (ie. a superstructure may span multiple adjacent regions of the target). [0040] For training purposes, in addition to feature vector input, the class's classifier parameters must be defined. [0041] FIG. 4 shows a block diagram for a predictive classification mapper employing a Gaussian mixture model neural network (GMM-NN) method. The parameters that need to be defined for the GMM-NN are the number of mixture components or centers in the model, the type of model, such as spherical, as well as its dimensionality. The Netlab neural network toolbox, part of the MATLAB™ package, available from The Mathworks, of Natick, Mass., may be used for the creation of the GMM with the ‘gmm’ function. GMM initialization then occurs by using the ‘gmminit’ function in the same toolbox. The expectation maximization (EM) algorithm is used to train the GMM and represent the class's input data as a mixture of weighted Gaussian components. [0042] For testing purposes, each class 410 in the database (6 in one embodiment), as well as a class of unknown targets of significantly larger size (referred to herein as an “unknown large” target type) and a class of unknown targets of significantly smaller size (referred to herein as an “unknown small” target type) are represented by a set of GMM parameters, generated using the expectation maximization algorithm.)
3. The method of claim 2, wherein when the particular AOI task of the first domain is the classification task, the task inspection sub-model is a multi-layer perceivorator (MLP) model; when the particular AOI task of the first domain is the segmentation task, the task inspection sub-model is a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism; and when the particular AOI task of the first domain is the identification task, the task inspection sub-model is a boundary frame decoder with a classification result of the CNN model or the decoder based on the attention mechanism. (Satheyendra: [0035] Feature extraction is important when distinguishing between classes. In one embodiment, the process begins by isolating the target region using a segmentation technique; edges are then detected using target lines found by Hough processing. [0039]-[0044], Figures 3-4, [0039] FIG. 3 represents the block diagram, in one embodiment, for the creation of a frame's feature vector. The Hough lines are initially used to isolate the target's center line in the image, its image line, and the refined length. The refined length estimate is formed by processing of the input sub-masked image using sum normalized range profile (SNRP) methodology. This length estimate is averaged (via median operator) with the previous 4 length determinations and is the first target feature. The purpose of the median operator is to avoid instantaneous erroneous fluctuations in the length estimates such as land returns and/or noise spikes. Using the locations of Hough lines and respective (line) designations, pertinent vertical lines are extracted. These lines are then compared with the locations of the peaks associated with the stored SNRP summed profile. If they occur in the same region, then the lines are designated as possible point of interest (POI) locations. If the Doppler extent is greater than a pre-determined threshold, the POI is deemed to be a possible reflector/rotator; otherwise it is deemed to be a possible mast/superstructure. Additional POI weighting occurs if high correlation exists with the SNRP peak locations. One range centroid of the POIs is stored for every target region; each such range centroid is then designated with a weight. If no POI exists in a region, the weighting is zero. The type designations for the POI are 0 for no POI, 1 for superstructure/mast, or 2 for rotator/reflector. The use of multiple regions may be advantageous in converting an otherwise 2-D feature extraction process into a 1-D process, in which the use of standard and more intricate classifiers can occur. This projection also speeds up the classification process. For larger targets, the use of more sections can also lead to a more refined POI range extent determination (ie. a superstructure may span multiple adjacent regions of the target). [0040] For training purposes, in addition to feature vector input, the class's classifier parameters must be defined. [0041] FIG. 4 shows a block diagram for a predictive classification mapper employing a Gaussian mixture model neural network (GMM-NN) method. The parameters that need to be defined for the GMM-NN are the number of mixture components or centers in the model, the type of model, such as spherical, as well as its dimensionality. The Netlab neural network toolbox, part of the MATLAB™ package, available from The Mathworks, of Natick, Mass., may be used for the creation of the GMM with the ‘gmm’ function. GMM initialization then occurs by using the ‘gmminit’ function in the same toolbox. The expectation maximization (EM) algorithm is used to train the GMM and represent the class's input data as a mixture of weighted Gaussian components. [0042] For testing purposes, each class 410 in the database (6 in one embodiment), as well as a class of unknown targets of significantly larger size (referred to herein as an “unknown large” target type) and a class of unknown targets of significantly smaller size (referred to herein as an “unknown small” target type) are represented by a set of GMM parameters, generated using the expectation maximization algorithm.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the fusion based classifier of Huang with the teachings of Sathyendra for data fusion analytics for automatic target recognition. The determination of obviousness is predicated upon the following findings: both references are in the same overall field of endeavor for learned fusion-based image recognition models. One skilled in the art would have been motivated to modify Huang’s fusion based classifier in order to enhance the use of data fusion analytics in the process for learned target recognition as described by Sathyendra. The improvement would enhance the overall accuracy, reliability and confidence in detecting different types of targets. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Huang, while the teaching of Sathyendra continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of automated target recognition (ATR) with improved reliability, confidence and accuracy. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 2.
The combination of Huang and Sathyendra teaches:
2. The method of claim 1, wherein the particular AOI task of the first domain comprises at least one or at least two of the following tasks: a classification task for classifying the image of the inspected component into one class of a plurality of classes; a segmentation task for segmenting a defective area in the image of the inspected component; and an identification task for identifying the defective area in the image of the inspected component. (Huang: [0039]-[0041], [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0041] After obtaining the first probability vector p1 (i.e., probability vector v2), the output layer 153 may receive the first probability vector p1, and determine the class of the input data IN according to the first probability vector p 1. In the embodiment, among the elements (V1,2, V2,2, V3,2, V4,2 and V5,2) of the first probability vector p1 (i.e., probability vector v2), the class 1 has the largest probability value (V1,2=0.781814). Based on this, the output layer 153 may classify the input data IN into class 1. Increasing the precision rate of a defect class may effectively reduce the false discovery rate (FDR) of the defect class. In this way, the number of samples of the product having the defect class may be reduced, thereby reducing the burden on the quality review personnel to perform manual re-examination of the sample. [0048] FIG. 2 is a schematic diagram of a classification system 10 based on probability fusion according to an embodiment of the disclosure. Classification system 10 is adapted to classifying the appearance defect of an item (e.g.: wafer) or the appearance defect of a printed circuit board into one of a variety of classes. The classification system 10 may include an automatic optical inspection equipment (or automatic visual inspection equipment) 110, a processor 130, and a classifier 150 (as shown in FIG. 1A). Automatic optical inspection (AOI) is a high-speed, high-accuracy optical image detection system that uses “mechanical vision” as an object to detect and replace human eye, brain or hand movements to detect the quality of the product or whether there are defects, etc. Automatic optical inspection technology is a non-contact detection technology that uses automatic optical inspection equipment to obtain the surface state of semi-finished or finished products (e.g.: wafers), and then uses image processing technology to detect defects such as foreign objects or pattern anomalies. Automatic optical inspection technology may improve the traditional drawbacks of using optical instruments for human detection. The automatic optical inspection equipment 110 is used to obtain image data of the appearance of a wafer or a printed circuit board. The image data may be used as the input data IN of the classifier 150. Satheyendra: [0035] Feature extraction is important when distinguishing between classes. In one embodiment, the process begins by isolating the target region using a segmentation technique; edges are then detected using target lines found by Hough processing. [0039]-[0044], Figures 3-4, [0039] FIG. 3 represents the block diagram, in one embodiment, for the creation of a frame's feature vector. The Hough lines are initially used to isolate the target's center line in the image, its image line, and the refined length. The refined length estimate is formed by processing of the input sub-masked image using sum normalized range profile (SNRP) methodology. This length estimate is averaged (via median operator) with the previous 4 length determinations and is the first target feature. The purpose of the median operator is to avoid instantaneous erroneous fluctuations in the length estimates such as land returns and/or noise spikes. Using the locations of Hough lines and respective (line) designations, pertinent vertical lines are extracted. These lines are then compared with the locations of the peaks associated with the stored SNRP summed profile. If they occur in the same region, then the lines are designated as possible point of interest (POI) locations. If the Doppler extent is greater than a pre-determined threshold, the POI is deemed to be a possible reflector/rotator; otherwise it is deemed to be a possible mast/superstructure. Additional POI weighting occurs if high correlation exists with the SNRP peak locations. One range centroid of the POIs is stored for every target region; each such range centroid is then designated with a weight. If no POI exists in a region, the weighting is zero. The type designations for the POI are 0 for no POI, 1 for superstructure/mast, or 2 for rotator/reflector. The use of multiple regions may be advantageous in converting an otherwise 2-D feature extraction process into a 1-D process, in which the use of standard and more intricate classifiers can occur. This projection also speeds up the classification process. For larger targets, the use of more sections can also lead to a more refined POI range extent determination (ie. a superstructure may span multiple adjacent regions of the target). [0040] For training purposes, in addition to feature vector input, the class's classifier parameters must be defined. [0041] FIG. 4 shows a block diagram for a predictive classification mapper employing a Gaussian mixture model neural network (GMM-NN) method. The parameters that need to be defined for the GMM-NN are the number of mixture components or centers in the model, the type of model, such as spherical, as well as its dimensionality. The Netlab neural network toolbox, part of the MATLAB™ package, available from The Mathworks, of Natick, Mass., may be used for the creation of the GMM with the ‘gmm’ function. GMM initialization then occurs by using the ‘gmminit’ function in the same toolbox. The expectation maximization (EM) algorithm is used to train the GMM and represent the class's input data as a mixture of weighted Gaussian components. [0042] For testing purposes, each class 410 in the database (6 in one embodiment), as well as a class of unknown targets of significantly larger size (referred to herein as an “unknown large” target type) and a class of unknown targets of significantly smaller size (referred to herein as an “unknown small” target type) are represented by a set of GMM parameters, generated using the expectation maximization algorithm.)
Consider Claim 3.
The combination of Huang and Sathyendra teaches:
3. The method of claim 2, wherein when the particular AOI task of the first domain is the classification task, the task inspection sub-model is a multi-layer perceivorator (MLP) model; when the particular AOI task of the first domain is the segmentation task, the task inspection sub-model is a mask decoder of a convolutional neural network (CNN) model or a decoder based on an attention mechanism; and when the particular AOI task of the first domain is the identification task, the task inspection sub-model is a boundary frame decoder with a classification result of the CNN model or the decoder based on the attention mechanism. (Satheyendra: [0035] Feature extraction is important when distinguishing between classes. In one embodiment, the process begins by isolating the target region using a segmentation technique; edges are then detected using target lines found by Hough processing. [0039]-[0044], Figures 3-4, [0039] FIG. 3 represents the block diagram, in one embodiment, for the creation of a frame's feature vector. The Hough lines are initially used to isolate the target's center line in the image, its image line, and the refined length. The refined length estimate is formed by processing of the input sub-masked image using sum normalized range profile (SNRP) methodology. This length estimate is averaged (via median operator) with the previous 4 length determinations and is the first target feature. The purpose of the median operator is to avoid instantaneous erroneous fluctuations in the length estimates such as land returns and/or noise spikes. Using the locations of Hough lines and respective (line) designations, pertinent vertical lines are extracted. These lines are then compared with the locations of the peaks associated with the stored SNRP summed profile. If they occur in the same region, then the lines are designated as possible point of interest (POI) locations. If the Doppler extent is greater than a pre-determined threshold, the POI is deemed to be a possible reflector/rotator; otherwise it is deemed to be a possible mast/superstructure. Additional POI weighting occurs if high correlation exists with the SNRP peak locations. One range centroid of the POIs is stored for every target region; each such range centroid is then designated with a weight. If no POI exists in a region, the weighting is zero. The type designations for the POI are 0 for no POI, 1 for superstructure/mast, or 2 for rotator/reflector. The use of multiple regions may be advantageous in converting an otherwise 2-D feature extraction process into a 1-D process, in which the use of standard and more intricate classifiers can occur. This projection also speeds up the classification process. For larger targets, the use of more sections can also lead to a more refined POI range extent determination (ie. a superstructure may span multiple adjacent regions of the target). [0040] For training purposes, in addition to feature vector input, the class's classifier parameters must be defined. [0041] FIG. 4 shows a block diagram for a predictive classification mapper employing a Gaussian mixture model neural network (GMM-NN) method. The parameters that need to be defined for the GMM-NN are the number of mixture components or centers in the model, the type of model, such as spherical, as well as its dimensionality. The Netlab neural network toolbox, part of the MATLAB™ package, available from The Mathworks, of Natick, Mass., may be used for the creation of the GMM with the ‘gmm’ function. GMM initialization then occurs by using the ‘gmminit’ function in the same toolbox. The expectation maximization (EM) algorithm is used to train the GMM and represent the class's input data as a mixture of weighted Gaussian components. [0042] For testing purposes, each class 410 in the database (6 in one embodiment), as well as a class of unknown targets of significantly larger size (referred to herein as an “unknown large” target type) and a class of unknown targets of significantly smaller size (referred to herein as an “unknown small” target type) are represented by a set of GMM parameters, generated using the expectation maximization algorithm. Huang: [0093] In an embodiment, the classification system of the disclosure may also gradually increase the number of applied neural networks. FIG. 13 is a schematic diagram of a plurality of neural network-based classification systems 1300 having a multi-crop architecture according to an embodiment of the disclosure. The classification system 1300 is suitable for users who are interested in a single class. The classification system 1300 may include an input layer 1310, a neural network 1320, a cropped layer 1330, a plurality of sub-classifiers (including: sub-classifiers 1511, 1512, 1513, 1514, and 1515), a fusion layer 152, and an output layer 153. For ease of explanation, all of the components (including sub-classifiers 1511, 1512, 1513, 1514, 1515, and the fusion layer 152) framed by block 13 are collectively referred to as first module 13.)
Consider Claim 4.
The combination of Huang and Sathyendra teaches:
4. The method of claim 1, wherein the generic feature extraction sub-model is a visual base model based on an attention mechanism. (Huang: 0035] Returning to FIG. 1A, the fusion layer 152 may select a first probability vector p1 having an extremum value corresponding to a first class-of-interest from the plurality of probability vectors according to the first class-of-interest, wherein the first class-of-interest corresponds to one of a plurality of elements in each of the plurality of probability vectors. [0036] Specifically, after the plurality of probability vectors v1, v2, . . . , and vn are generated in the sub-classifier 151, the fusion layer 152 may obtain an element corresponding to the first class-of-interest in each probability vector (it is assumed below that each probability vector has M elements, and each of the M elements corresponds to a different class, where element k corresponds to first class-of-interest). Next, the fusion layer 152 may select an element k having an extremum value from among n element k corresponding to n probability vectors (i.e., probability vector v1, v2, . . . , and vn), where k is one of 1 to n. The probability vector corresponding to the element k with the extremum value may be used as first probability vector p1. In the disclosure, the extremum value may represent a maximum value and a minimum value. For example, suppose the classifier 150 in FIG. 1A classifies the input data IN into one of five classes (i.e.: M=5), and the input data IN includes five pieces of data (i.e.: n=5). For the data i1, i2, i3, i4, and i5, the sub-classifier 151 may generate 5 probability vector v1, v2, v3, v4, and v5 according to the data, and the 5 probability vectors may be expressed as an example of a probability matrix of equation (1),)
Consider Claim 5.
The combination of Huang and Sathyendra teaches:
5. The method of claim 4, wherein the generic feature extraction sub-model is trained based on images of industrial components in a plurality of domains, and wherein the plurality of domains correspond to a plurality of industrial component types, respectively or correspond to a plurality of industrial component production lines, respectively. (Huang: [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0062] The output layer 453 may output an output data OUT representing a class of input data IN. After obtaining the third probability vector p3, the output layer 453 may receive the third probability vector p3, and the class of the input data IN is determined according to the third probability vector p3. For example, in the embodiment, the element of the third row of the third probability vector p3 has the largest probability value (0.657510). Based on this, the output layer 453 classifies the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the class output by the output layer 453 may be, for example, a type of appearance defect of the wafer or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto.)
Consider Claim 6.
The combination of Huang and Sathyendra teaches:
6. The method of claim 5, wherein the first domain is one domain in the plurality of domains, or the first domain is one domain other than the plurality of domains, and wherein the task inspection sub-model is trained based on an image of an industrial component in the first domain. (Huang: [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0062] The output layer 453 may output an output data OUT representing a class of input data IN. After obtaining the third probability vector p3, the output layer 453 may receive the third probability vector p3, and the class of the input data IN is determined according to the third probability vector p3. For example, in the embodiment, the element of the third row of the third probability vector p3 has the largest probability value (0.657510). Based on this, the output layer 453 classifies the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the class output by the output layer 453 may be, for example, a type of appearance defect of the wafer or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto.)
Consider Claim 7.
The combination of Huang and Sathyendra teaches:
7. The method of claim 1, further comprising: obtaining a second image of a second inspected component of a second domain, the second domain corresponding to a second industrial component type or corresponding to a second industrial component production line; inputting the second image of the second inspected component into the inspection model, the inspection model further comprising at least one second task inspection sub-model associated with a particular AOI task of the second domain; generating, by the generic feature extraction sub-model, a first feature representation of the second image based on the second image of the second inspected component; and generating, by the at least one second task inspection sub-model, at least one inspection result associated with the particular AOI task of the second domain based on the first feature representation of the second image. (Huang: [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0062] The output layer 453 may output an output data OUT representing a class of input data IN. After obtaining the third probability vector p3, the output layer 453 may receive the third probability vector p3, and the class of the input data IN is determined according to the third probability vector p3. For example, in the embodiment, the element of the third row of the third probability vector p3 has the largest probability value (0.657510). Based on this, the output layer 453 classifies the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the class output by the output layer 453 may be, for example, a type of appearance defect of the wafer or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. [0071] FIG. 7 is a schematic diagram of applying a classifier 450 to a classification system 700 based on a multi-crop (12-crop) neural network according to an embodiment of the disclosure. [0072] In the field of neural network-based image recognition, the multi-crop evaluation technique may be performed by cropping a single image into a plurality of cropped parts, and respectively inputs the plurality of cropped parts into the corresponding classifier to increase the diversity of the label data. In FIG. 7, the classification system 700 may crop the image data IM and the mirror data of the image data IM into six cropped parts, respectively (a total of 12 cropped parts). For example, the classification system may crop the upper right corner of the image data IM into an upper right corner cropped part, or crop the upper left corner of the image data IM into an upper left corner cropped part (the size of the cropped part may be smaller than the size of the image data IM), etc., but the disclosure is not limited thereto. Satheyendra: [0004] Inverse synthetic aperture radar (ISAR) is a signal processing technique used to form a two-dimensional (2-D) radar image from moving target objects by separating radar returns in Doppler frequency and in range. ISAR is possible with or without radar platform motion. An ISAR 2-D image is comprised of different intensity pixels of reflected point scatterers located at particular range and Doppler bin indices. [0021] According to an embodiment of the present invention there is provided a system for automatic target recognition of a target, the system including a processing unit configured to: receive a sequence of imaging radar images of the target; form a feature vector including measured characteristics of the target; perform a first target recognition attempt, the performing of the first target recognition attempt including: using a Gaussian mixture model neural network classifier to generate a first plurality of probability likelihoods, each of the first plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a first set of class designation rules to produce a first class designation, the first class designation corresponding to one of the plurality of candidate target types; perform a second target recognition attempt, the performing of the second target recognition attempt including: using a radial basis function neural network classifier to generate a second plurality of probability likelihoods, each of the second plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a second set of class designation rules to produce a second class designation, the second class designation corresponding to one of the plurality of candidate target types; perform a third target recognition attempt, the performing of the third target recognition attempt including: using a vector quantization classifier to generate a third plurality of probability likelihoods, each of the third plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a third set of class designation rules to produce a third class designation, the third class designation corresponding to one of the plurality of candidate target types; and combine the first class designation, the second class designation and the third class designation to generate an overall class designation. [0036] FIG. 1 represents an overall block diagram for an ATR engine process according to one embodiment. Data inputs include radar data 110 and the trained classifier parameters 115, which are fed into the signal processing block 120 to attain the target designations for the radar data 110. Off-line training is used to determine the trained classifier parameters 115 that can then be used in real time to determine the class designation for a given input image. The signal processing block includes an image formation block 125 that creates the input 2-D image. The second step is to determine, in a good frame selection block 130, the suitability of the input image for the classification problem. If the image is deemed suitable, a feature vector is extracted from the image in a multi-frames features block 135. A stored history of one-dimensional (1-D) features from previous frames also aids in this feature vector extraction process. The extracted features together act as inputs for the predictive classification mapper 140 that determines a target class.)
Consider Claim 9.
The combination of Huang and Sathyendra teaches:
9. The method of claim 8, wherein training the generic feature extraction sub-model in the inspection model based on the multi-domain AOI training dataset comprises: performing data enhancement on the multi-domain AOI training dataset without a label to obtain a multi-domain AOI training image pair set, and training the generic feature extraction sub-model using a comparative learning method based on the multi-domain AOI training image pair set; and/or training the generic feature extraction sub-model using a masked autoencoder (MAE) method based on the multi-domain AOI training dataset without the label. (Satheyendra: [0038] FIG. 2 represents the top level block diagram for the frame's feature extraction algorithms, wherein the dashed area, representing the multi-frames features block 135, repeats for each frame. The isolate target region block 210 isolates a rough silhouette of the target region, which acts as a submask in conjunction with the input image to form the input to the length estimation and Hough processing block 215. The target length is estimated and the target range region is further refined, and acts as input to the Hough processing algorithms. The Hough peaks and Hough lines for the targets are extracted, as well as a refined target length estimate, which act as input to the feature vector generation block 220. The feature vector generation block 220 constructs the frame's feature vector, which is used for classifier training and for testing. Training occurs off line and with the multi-frames features block 135 processing repeating for each training frame. Training may be performed with real radar data, e.g., ISAR data obtained in the field, to avoid the disadvantages associated with using simulated data for training. A set of real data may then be separated into a subset used for training, and a second subset used for testing, to assess the performance of the ATR system after training. Testing occurs in real time and for a single frame, which is processed through the full ATR engine, until a class designation is determined. [0039] FIG. 3 represents the block diagram, in one embodiment, for the creation of a frame's feature vector. The Hough lines are initially used to isolate the target's center line in the image, its image line, and the refined length. The refined length estimate is formed by processing of the input sub-masked image using sum normalized range profile (SNRP) methodology. This length estimate is averaged (via median operator) with the previous 4 length determinations and is the first target feature. The purpose of the median operator is to avoid instantaneous erroneous fluctuations in the length estimates such as land returns and/or noise spikes. Using the locations of Hough lines and respective (line) designations, pertinent vertical lines are extracted. These lines are then compared with the locations of the peaks associated with the stored SNRP summed profile. If they occur in the same region, then the lines are designated as possible point of interest (POI) locations. If the Doppler extent is greater than a pre-determined threshold, the POI is deemed to be a possible reflector/rotator; otherwise it is deemed to be a possible mast/superstructure. Additional POI weighting occurs if high correlation exists with the SNRP peak locations. One range centroid of the POIs is stored for every target region; each such range centroid is then designated with a weight. If no POI exists in a region, the weighting is zero. The type designations for the POI are 0 for no POI, 1 for superstructure/mast, or 2 for rotator/reflector. The use of multiple regions may be advantageous in converting an otherwise 2-D feature extraction process into a 1-D process, in which the use of standard and more intricate classifiers can occur. This projection also speeds up the classification process. For larger targets, the use of more sections can also lead to a more refined POI range extent determination (ie. a superstructure may span multiple adjacent regions of the target). [0040] For training purposes, in addition to feature vector input, the class's classifier parameters must be defined. Embodiments of the present invention encompass the Gaussian mixture model neural network (GMM-NN), the radial basis functions neural network (RBF-NN), the vector quantizer (VQ), and a classifier fusion method, which combines the class designations produced by all of these techniques.)
Consider Claim 10.
The combination of Huang and Sathyendra teaches:
10. The method of claim 9, wherein: training the generic feature extraction sub-model using the comparative learning method based on the multi-domain AOI training image pair set comprises: receiving, by the generic feature extraction sub-model, an image pair in the multi-domain AOI training image pair set and outputting a first feature representation of the image pair, obtaining a similarity of the image pair by a comparative learning module based on the first feature representation of the image pair, obtaining a comparative learning loss based on the similarity of the image pair, and updating the generic feature extraction sub-model based on the comparative learning loss; and/or training the generic feature extraction sub-model using the MAE method comprises: masking an original image in the multi-domain AOI training dataset to obtain a masked image, receiving, by the generic feature extraction sub-model, the masked image and outputting a first feature representation of the masked image, predicting, by a decoder module, a restored image based on the first feature representation of the masked image, obtaining an MAE loss based on the restored image and the original image, and updating the generic feature extraction sub-model based on the MAE loss. (Satheyendra: [0038] FIG. 2 represents the top level block diagram for the frame's feature extraction algorithms, wherein the dashed area, representing the multi-frames features block 135, repeats for each frame. The isolate target region block 210 isolates a rough silhouette of the target region, which acts as a submask in conjunction with the input image to form the input to the length estimation and Hough processing block 215. The target length is estimated and the target range region is further refined, and acts as input to the Hough processing algorithms. The Hough peaks and Hough lines for the targets are extracted, as well as a refined target length estimate, which act as input to the feature vector generation block 220. The feature vector generation block 220 constructs the frame's feature vector, which is used for classifier training and for testing. Training occurs off line and with the multi-frames features block 135 processing repeating for each training frame. Training may be performed with real radar data, e.g., ISAR data obtained in the field, to avoid the disadvantages associated with using simulated data for training. A set of real data may then be separated into a subset used for training, and a second subset used for testing, to assess the performance of the ATR system after training. Testing occurs in real time and for a single frame, which is processed through the full ATR engine, until a class designation is determined. [0039] FIG. 3 represents the block diagram, in one embodiment, for the creation of a frame's feature vector. The Hough lines are initially used to isolate the target's center line in the image, its image line, and the refined length. The refined length estimate is formed by processing of the input sub-masked image using sum normalized range profile (SNRP) methodology. This length estimate is averaged (via median operator) with the previous 4 length determinations and is the first target feature. The purpose of the median operator is to avoid instantaneous erroneous fluctuations in the length estimates such as land returns and/or noise spikes. Using the locations of Hough lines and respective (line) designations, pertinent vertical lines are extracted. These lines are then compared with the locations of the peaks associated with the stored SNRP summed profile. If they occur in the same region, then the lines are designated as possible point of interest (POI) locations. If the Doppler extent is greater than a pre-determined threshold, the POI is deemed to be a possible reflector/rotator; otherwise it is deemed to be a possible mast/superstructure. Additional POI weighting occurs if high correlation exists with the SNRP peak locations. One range centroid of the POIs is stored for every target region; each such range centroid is then designated with a weight. If no POI exists in a region, the weighting is zero. The type designations for the POI are 0 for no POI, 1 for superstructure/mast, or 2 for rotator/reflector. The use of multiple regions may be advantageous in converting an otherwise 2-D feature extraction process into a 1-D process, in which the use of standard and more intricate classifiers can occur. This projection also speeds up the classification process. For larger targets, the use of more sections can also lead to a more refined POI range extent determination (ie. a superstructure may span multiple adjacent regions of the target). [0040] For training purposes, in addition to feature vector input, the class's classifier parameters must be defined. Embodiments of the present invention encompass the Gaussian mixture model neural network (GMM-NN), the radial basis functions neural network (RBF-NN), the vector quantizer (VQ), and a classifier fusion method, which combines the class designations produced by all of these techniques.)
Consider Claim 11.
The combination of Huang and Sathyendra teaches:
11. The method of claim 8, wherein training at least one task inspection sub-model associated with the particular AOI task of the first domain in the inspection model based on the image of the industrial component in the first domain as the single-domain AOI training dataset comprises: training the task inspection sub-model based on the single-domain AOI training dataset with a label. (Satheyendra: [0042] For testing purposes, each class 410 in the database (6 in one embodiment), as well as a class of unknown targets of significantly larger size (referred to herein as an “unknown large” target type) and a class of unknown targets of significantly smaller size (referred to herein as an “unknown small” target type) are represented by a set of GMM parameters, generated using the expectation maximization algorithm. The classes in the database are referred to herein as Class 1 through Class 6, or as “CGCPS”, “TRB6”, TWR823”, “CGCT”, “LA”, and “MII”; these 6 classes and the “unknown small” and “unknown classes” are labeled “GMM Class 1” through “GMM class 8” in FIG. 4. When confronted with a test feature vector the MATLAB™ Netlab toolbox function ‘gmmprob’ is used to find the probability associated with each of the 8 GMM models (including the 6 classes in the database, and the unknown large and unknown small target types). The ensuing maximization is used as a reference for class designations. If a class designation is deemed applicable, then the maximization of the gmm probability corresponds to the class. Huang: [0071] FIG. 7 is a schematic diagram of applying a classifier 450 to a classification system 700 based on a multi-crop (12-crop) neural network according to an embodiment of the disclosure. [0072] In the field of neural network-based image recognition, the multi-crop evaluation technique may be performed by cropping a single image into a plurality of cropped parts, and respectively inputs the plurality of cropped parts into the corresponding classifier to increase the diversity of the label data. In FIG. 7, the classification system 700 may crop the image data IM and the mirror data of the image data IM into six cropped parts, respectively (a total of 12 cropped parts). For example, the classification system may crop the upper right corner of the image data IM into an upper right corner cropped part, or crop the upper left corner of the image data IM into an upper left corner cropped part (the size of the cropped part may be smaller than the size of the image data IM), etc., but the disclosure is not limited thereto.)
Consider Claim 12.
The combination of Huang and Sathyendra teaches:
12. The method of claim 8, further comprising: training at least one second task inspection sub-model associated with a second particular AOI task of a second domain in the inspection model based on an image of an industrial component in the second region as a second single-domain AOI training dataset, the second domain corresponding to a second industrial component type or corresponding to a second industrial component production line, wherein the trained generic feature extraction sub-model receives a second image in the second single-domain AOI training dataset and outputs a first feature representation of the second image, and the at least one second task inspection sub-model receives the first feature representation of the second image and predicts at least one inspection result associated with the second particular AOI task of the second domain. (Huang: [0039] The output layer 153 may output an output data OUT of a class representing the input data IN. Specifically, after obtaining the first probability vector pl (i.e., probability vector vl), the output layer 153 may receive the first probability vector p1 and according to the first probability vector pl to determine the class of the input data IN. In the embodiment, among the elements (V1,1, V2,1, V3,1, V4,1, and V5,1) of the first probability vector p1 (i.e., probability vector v1), V3,1 representing class 3 has the largest probability value (V3,1=0.569379). Based on this, the output layer 153 may classify the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the output data OUT outputted by the output layer 153 may be, for example, a type of appearance defect of the wafer to be inspected, or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. Taking the wafer manufacturing industry as an example, the output data OUT may represent classes including: scratch, arrow, particle, discoloration, normal, etc. defect class. When a defect class can cause serious problems, reducing the miss rate of the defect class may effectively prevent the product having the defect of the class from flowing into the market. [0062] The output layer 453 may output an output data OUT representing a class of input data IN. After obtaining the third probability vector p3, the output layer 453 may receive the third probability vector p3, and the class of the input data IN is determined according to the third probability vector p3. For example, in the embodiment, the element of the third row of the third probability vector p3 has the largest probability value (0.657510). Based on this, the output layer 453 classifies the input data IN into class 3. In the industry of wafer fabrication or printed circuit board manufacturing, the class output by the output layer 453 may be, for example, a type of appearance defect of the wafer or a type of appearance defect of the printed circuit board, but the disclosure is not limited thereto. [0071] FIG. 7 is a schematic diagram of applying a classifier 450 to a classification system 700 based on a multi-crop (12-crop) neural network according to an embodiment of the disclosure. [0072] In the field of neural network-based image recognition, the multi-crop evaluation technique may be performed by cropping a single image into a plurality of cropped parts, and respectively inputs the plurality of cropped parts into the corresponding classifier to increase the diversity of the label data. In FIG. 7, the classification system 700 may crop the image data IM and the mirror data of the image data IM into six cropped parts, respectively (a total of 12 cropped parts). For example, the classification system may crop the upper right corner of the image data IM into an upper right corner cropped part, or crop the upper left corner of the image data IM into an upper left corner cropped part (the size of the cropped part may be smaller than the size of the image data IM), etc., but the disclosure is not limited thereto. Satheyendra: [0004] Inverse synthetic aperture radar (ISAR) is a signal processing technique used to form a two-dimensional (2-D) radar image from moving target objects by separating radar returns in Doppler frequency and in range. ISAR is possible with or without radar platform motion. An ISAR 2-D image is comprised of different intensity pixels of reflected point scatterers located at particular range and Doppler bin indices. [0021] According to an embodiment of the present invention there is provided a system for automatic target recognition of a target, the system including a processing unit configured to: receive a sequence of imaging radar images of the target; form a feature vector including measured characteristics of the target; perform a first target recognition attempt, the performing of the first target recognition attempt including: using a Gaussian mixture model neural network classifier to generate a first plurality of probability likelihoods, each of the first plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a first set of class designation rules to produce a first class designation, the first class designation corresponding to one of the plurality of candidate target types; perform a second target recognition attempt, the performing of the second target recognition attempt including: using a radial basis function neural network classifier to generate a second plurality of probability likelihoods, each of the second plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a second set of class designation rules to produce a second class designation, the second class designation corresponding to one of the plurality of candidate target types; perform a third target recognition attempt, the performing of the third target recognition attempt including: using a vector quantization classifier to generate a third plurality of probability likelihoods, each of the third plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a third set of class designation rules to produce a third class designation, the third class designation corresponding to one of the plurality of candidate target types; and combine the first class designation, the second class designation and the third class designation to generate an overall class designation. [0036] FIG. 1 represents an overall block diagram for an ATR engine process according to one embodiment. Data inputs include radar data 110 and the trained classifier parameters 115, which are fed into the signal processing block 120 to attain the target designations for the radar data 110. Off-line training is used to determine the trained classifier parameters 115 that can then be used in real time to determine the class designation for a given input image. The signal processing block includes an image formation block 125 that creates the input 2-D image. The second step is to determine, in a good frame selection block 130, the suitability of the input image for the classification problem. If the image is deemed suitable, a feature vector is extracted from the image in a multi-frames features block 135. A stored history of one-dimensional (1-D) features from previous frames also aids in this feature vector extraction process. The extracted features together act as inputs for the predictive classification mapper 140 that determines a target class.)
Consider Claim 14.
The combination of Huang and Sathyendra teaches:
14. The device of claim 13, wherein the image obtaining module is configured to obtain a second image of a second inspected component of a second domain, the second domain corresponding to a second industrial component type or corresponding to a second industrial component production line, wherein the inspection module further comprises at least one second task inspection sub-module associated with a particular AOI task of the second domain, wherein the generic feature extraction sub-module is configured to generate a first feature representation of the second image based on the second image of the second inspected component, and the at least one second task inspection sub-module is configured to generate at least one inspection result associated with the particular AOI task of the second domain based on the first feature representation of the second image. (Satheyendra: [0004] Inverse synthetic aperture radar (ISAR) is a signal processing technique used to form a two-dimensional (2-D) radar image from moving target objects by separating radar returns in Doppler frequency and in range. ISAR is possible with or without radar platform motion. An ISAR 2-D image is comprised of different intensity pixels of reflected point scatterers located at particular range and Doppler bin indices. [0021] According to an embodiment of the present invention there is provided a system for automatic target recognition of a target, the system including a processing unit configured to: receive a sequence of imaging radar images of the target; form a feature vector including measured characteristics of the target; perform a first target recognition attempt, the performing of the first target recognition attempt including: using a Gaussian mixture model neural network classifier to generate a first plurality of probability likelihoods, each of the first plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a first set of class designation rules to produce a first class designation, the first class designation corresponding to one of the plurality of candidate target types; perform a second target recognition attempt, the performing of the second target recognition attempt including: using a radial basis function neural network classifier to generate a second plurality of probability likelihoods, each of the second plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a second set of class designation rules to produce a second class designation, the second class designation corresponding to one of the plurality of candidate target types; perform a third target recognition attempt, the performing of the third target recognition attempt including: using a vector quantization classifier to generate a third plurality of probability likelihoods, each of the third plurality of probability likelihoods corresponding to one of a plurality of candidate target types; and using a third set of class designation rules to produce a third class designation, the third class designation corresponding to one of the plurality of candidate target types; and combine the first class designation, the second class designation and the third class designation to generate an overall class designation. [0036] FIG. 1 represents an overall block diagram for an ATR engine process according to one embodiment. Data inputs include radar data 110 and the trained classifier parameters 115, which are fed into the signal processing block 120 to attain the target designations for the radar data 110. Off-line training is used to determine the trained classifier parameters 115 that can then be used in real time to determine the class designation for a given input image. The signal processing block includes an image formation block 125 that creates the input 2-D image. The second step is to determine, in a good frame selection block 130, the suitability of the input image for the classification problem. If the image is deemed suitable, a feature vector is extracted from the image in a multi-frames features block 135. A stored history of one-dimensional (1-D) features from previous frames also aids in this feature vector extraction process. The extracted features together act as inputs for the predictive classification mapper 140 that determines a target class. Huang: [0056] The classifier 450 may include a sub-classifier 451, a fusion layer 452, a second fusion layer 454, and an output layer 453. In this embodiment, it is assumed that input data IN includes n pieces of data such as data i1, data i2, . . . , and data in. The sub-classifier 451 may generate a plurality of probability vectors v1, v2, . . . , and vn according to the input data IN. The sub-classifier 451 generates a probability vector in a similar manner to the sub-classifier 151, and will not be repeated here. [0068] FIG. 6B further illustrates a flowchart of step S604 according to an embodiment of the disclosure. In some embodiments, step S604 may be broken down into step S6041, step S6042, and step S6043. In step S6041, the output layer 453 may perform a likelihood ratio test on the third probability vector p3 according to the first class-of-interest and/or the second class-of-interest. If the result of the likelihood ratio test is true (i.e.: equation (2) is true), then proceeding to step S6042, the output layer 453 may classify the input data IN into the first class-of-interest and/or the second class-of-interest. On the other hand, if the result of the likelihood ratio test is false (i.e.: equation (2) is false), then the process proceeds to step S6043, the output layer 153 classifies the input data IN as a non-first class-of-interest and/or a non-second class-of-interest. [0069] In an embodiment, after proceeding to step S6043, if a likelihood ratio test performed on the third probability vector p3 according to the first class-of-interest by the output layer 453 is true, and a likelihood ratio test perfomied on the third probability vector p3 according to the second class-of-interest by the output layer 453 is true, the output layer 153 may classify the input data IN into the class-of-interest having the higher probability between the first class-of-interest and second class-of-interest. For example, when a likelihood ratio test performed on the third probability vector p3 according to the first class-of-interest by the output layer 453 is true, and when a likelihood ratio test performed on the third probability vector p3 according to the second class-of-interest by the output layer 453 is true, if the probability that the output data IN belongs to the first class-of-interest is higher than the probability that the output data IN belongs to the second class-of-interest, the output layer 453 classifies the output data IN as the first class-of-interest.)
Consider Claim 15.
The combination of Huang and Sathyendra teaches:
15. A system for automatic optical inspection (AOI), comprising: an image capture apparatus configured to capture an image of an inspected component; one or more processors; and one or more memories, the memories having computer-executable instructions stored thereon, and the instructions, when run by the one or more processors, perform the operations of claim 1. (Huang: [0030] FIGS. 1 A is schematic diagrams of a fusion-based classifier 150 according to an embodiment of the disclosure, wherein the classifier 150 is adapted to classify input data into one of a plurality of classes, and the classifier 150 may be implemented by a hardware (e.g.: a circuit or an integrated circuit) or a software (e.g.: one or more modules stored in a storage medium), and the disclosure is not limited thereto. [0049] The processor 130 is coupled to the automatic optical inspection equipment (or the automatic visual inspection equipment) 110 and the classifier 150, and may forward the image data from the automatic optical inspection equipment (or the automatic visual inspection equipment) 110 to the classifier 150. The processor 130 may be, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor, digital signal processor (DSP), programmable controller, the application specific integrated circuit (ASIC) or other similar components or a combination of the above components, the disclosure is not limited thereto. The processor 130 may be used to access a plurality of modules in the storage medium. [0050] The classifier 150 may include a sub-classifier 151, a fusion layer 152, and an output layer 153. Since the function of the classifier 150 has been disclosed in the embodiment of FIG. 1A, it will not be described herein. Sathyendra:[0055] Elements of embodiments of the present invention may be implemented using one or more processing units. Processing unit hardware may include, for example, application specific integrated circuits (ASICs), general purpose or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). The term “processing unit” is used herein to include any combination of hardware, firmware, and software, employed to process data or digital signals. In a processing unit, as used herein, each function is performed either by hardware configured, i.e., hard-wired, to perform that function, or by more general purpose hardware, such as a CPU, configured to execute instructions stored in a non-transitory storage medium.)
Consider Claim 16.
The combination of Huang and Sathyendra teaches:
16. A computer system, comprising: one or more processors; and one or more memories, the memories having computer-executable instructions stored thereon, and the instructions, when run by the one or more processors, perform the operations of claim 1. (Huang: [0030] FIGS. 1 A is schematic diagrams of a fusion-based classifier 150 according to an embodiment of the disclosure, wherein the classifier 150 is adapted to classify input data into one of a plurality of classes, and the classifier 150 may be implemented by a hardware (e.g.: a circuit or an integrated circuit) or a software (e.g.: one or more modules stored in a storage medium), and the disclosure is not limited thereto. [0049] The processor 130 is coupled to the automatic optical inspection equipment (or the automatic visual inspection equipment) 110 and the classifier 150, and may forward the image data from the automatic optical inspection equipment (or the automatic visual inspection equipment) 110 to the classifier 150. The processor 130 may be, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor, digital signal processor (DSP), programmable controller, the application specific integrated circuit (ASIC) or other similar components or a combination of the above components, the disclosure is not limited thereto. The processor 130 may be used to access a plurality of modules in the storage medium. [0050] The classifier 150 may include a sub-classifier 151, a fusion layer 152, and an output layer 153. Since the function of the classifier 150 has been disclosed in the embodiment of FIG. 1A, it will not be described herein. Sathyendra:[0055] Elements of embodiments of the present invention may be implemented using one or more processing units. Processing unit hardware may include, for example, application specific integrated circuits (ASICs), general purpose or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). The term “processing unit” is used herein to include any combination of hardware, firmware, and software, employed to process data or digital signals. In a processing unit, as used herein, each function is performed either by hardware configured, i.e., hard-wired, to perform that function, or by more general purpose hardware, such as a CPU, configured to execute instructions stored in a non-transitory storage medium.)
Consider Claim 17.
The combination of Huang and Sathyendra teaches:
17. A machine-readable storage medium having executable instructions stored thereon, the instructions, when executed, cause one or more processors to perform the method of claim 1. (Huang: [0030] FIGS. 1 A is schematic diagrams of a fusion-based classifier 150 according to an embodiment of the disclosure, wherein the classifier 150 is adapted to classify input data into one of a plurality of classes, and the classifier 150 may be implemented by a hardware (e.g.: a circuit or an integrated circuit) or a software (e.g.: one or more modules stored in a storage medium), and the disclosure is not limited thereto. [0049] The processor 130 is coupled to the automatic optical inspection equipment (or the automatic visual inspection equipment) 110 and the classifier 150, and may forward the image data from the automatic optical inspection equipment (or the automatic visual inspection equipment) 110 to the classifier 150. The processor 130 may be, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor, digital signal processor (DSP), programmable controller, the application specific integrated circuit (ASIC) or other similar components or a combination of the above components, the disclosure is not limited thereto. The processor 130 may be used to access a plurality of modules in the storage medium. [0050] The classifier 150 may include a sub-classifier 151, a fusion layer 152, and an output layer 153. Since the function of the classifier 150 has been disclosed in the embodiment of FIG. 1A, it will not be described herein. Sathyendra:[0055] Elements of embodiments of the present invention may be implemented using one or more processing units. Processing unit hardware may include, for example, application specific integrated circuits (ASICs), general purpose or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). The term “processing unit” is used herein to include any combination of hardware, firmware, and software, employed to process data or digital signals. In a processing unit, as used herein, each function is performed either by hardware configured, i.e., hard-wired, to perform that function, or by more general purpose hardware, such as a CPU, configured to execute instructions stored in a non-transitory storage medium.)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image2.png
175
928
media_image2.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
August 6, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights