DETAILED ACTION
Response to Arguments
The amendments filed 8/10/2026 have been entered and made of record.
Applicant's amendments and corresponding arguments filed 8/10/2026 have been fully considered, but are not persuasive:
Re Claim 1, first, regarding to the limitation “for each training epoch, determining, from the defect probability matrices associated with said training epoch, a defect detection curve as a function of false alarms”,
Applicant states that BOUSSAID’s “the failed computer modeling” cited in the last Non-Final Office Action of 5/8/2026, is a translation error, should be “the adjusted computer modeling”, and therefore, is not “false alarms” in defect detection; thus, the cited references do not disclose “determining, a defect detection curve as a function of false alarms”;
However, the Examiner disagrees, because:
BADANES discloses a defect detection curve as a function of false alarms (see BADANES: e.g., Fig. 4, and, -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.
[0072] Capture rate (also referred to herein as recall) refers to the ratio of the captured defects over the whole defect population. False alarm rate refers to the ratio of the declared nuisance over the total declared defects. Precision refers to the ratio of the number of DOI over the total declared defects. It is noted that the terms capture rate, recall, false alarm rate, precision etc. are known terminologies in the art and should be understood in its broadest interpretation.--, in [0071]-[0072], and, -- [0082] According to certain embodiments, after processing an image, the supervised model 303 and unsupervised model 304 output the first grade map and the second grade map respectively. A respective threshold (denoted as Th.sub.s to be used for the supervised model, and Th.sub.us to be used for the unsupervised model, as illustrated in FIG. 4) can be applied to each grade map in order to generate a defect detection output, e.g., a defect map (denoted as Output.sub.s for the supervised model, and Output.sub.us for the unsupervised model, as illustrated in FIG. 4). The thresholds can be initially selected during the separate training phase as described above with reference to FIG. 3. According to certain embodiments of the present disclosure, the initially selected threshold of each (or at least one) model can be tuned so as to optimize the capture rate and false alarm rate of the model. In this case, the optimized parameters used for the combination, as mentioned above, are the optimized threshold(s), which can be applied to the corresponding grade map to obtain an optimized defect map. The optimized defect maps of both models can be combined to generate an overall defect detection result 405.
[0083] Specifically, in such cases, the validation set can be used to tune the threshold of one or both of the models according to user-defined capture rate and/or false alarm rate 406. By way of example, a validation image can be processed by both models. If an expected capture rate is pre-defined by the user (e.g., 90%), the thresholds of both models can be increased so as to minimize the false alarm rates, as long as the user-defined capture rate is not compromised. On the other hand, if an expected false alarm rate is pre-defined by the user, the thresholds of both models can be decreased so as to increase the capture rate, as long as the user-defined false rate is not compromised. A graph 408 illustrating the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) is shown in FIG. 4. As illustrated, a high performance model would have a high true positive rate and a low false positive rate, which would be within the top left target area. This means that most predictions of the model are consistent with the ground truth, and most of the declared defects are detected.--, in [0082]-[0083]);
BOUSSAID (as modified by GUO) and BADANES are combinable as they are in the same field of endeavor: defects detection based on machine learning and image processing. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify BOUSSAID (as modified by GUO)’s method using BADANES’s teachings by including a defect detection curve as a function of false alarms to BOUSSAID (as modified by GUO) ’s model evaluation in order to validate the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) (see BADANES: e.g., in [0071]-[0072], and [0082]-[0083]);
Applicant also states that above combination of BOUSSAID (as modified by GUO) and BADANES is not proper,
However, the Examiner disagrees, because, as pointed out and further highlighted above in cited BADANES’s disclosures:
BOUSSAID (as modified by GUO) and BADANES are defects detection based on machine learning and image processing, and as Applicant admits that BOUSSAID’s computer modeling applied in defect detections, particularly for aeronautical parts including contour of parts is adjusted, and/or readjusted in machine learning training, it is pointed in the Office Action, BOUSSAID’s adjusted computer modeling is evaluated during the validation of the defect detection, so that BOUSSAID’s adjusted computer modeling is a factor of the defect detection which affects the performance of the defect detection function; and
BADANES clearly teaches/discloses: -- False alarm rate refers to the ratio of the declared nuisance over the total declared defects. Precision refers to the ratio of the number of DOI over the total declared defects. It is noted that the terms capture rate, recall, false alarm rate, precision etc. are known terminologies in the art and should be understood in its broadest interpretation.--, in [0071]-[0072];
which is aligned with, and read on claimed ““determining, a defect detection curve as a function of false alarms”; and
the motivation and obviousness are well and properly established to further modify BOUSSAID (as modified by GUO)’s method using BADANES’s teachings by including a defect detection curve as a function of false alarms to BOUSSAID (as modified by GUO) ’s model evaluation in order to validate the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) (see BADANES: e.g., in [0071]-[0072], and [0082]-[0083]);
And, Applicant further asserts (in page 10 of the response) that Badanes selects a threshold, not a training epoch;
However, the Examiner disagrees, because:
Badanes teaches of the thresholds is applied to select suitable training epoch with higher accuracy (higher performance of the models) in the validation of defect detection models (see Badanes: e.g., --[0083] Specifically, in such cases, the validation set can be used to tune the threshold of one or both of the models according to user-defined capture rate and/or false alarm rate 406. By way of example, a validation image can be processed by both models. If an expected capture rate is pre-defined by the user (e.g., 90%), the thresholds of both models can be increased so as to minimize the false alarm rates, as long as the user-defined capture rate is not compromised. On the other hand, if an expected false alarm rate is pre-defined by the user, the thresholds of both models can be decreased so as to increase the capture rate, as long as the user-defined false rate is not compromised. A graph 408 illustrating the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) is shown in FIG. 4. As illustrated, a high performance model would have a high true positive rate and a low false positive rate, which would be within the top left target area. This means that most predictions of the model are consistent with the ground truth, and most of the declared defects are detected.--, in [0082]-[0083]);
And, BADANES discloses “supervisely training…” by supervised model 303, as well as unsupervised model 304.
Therefore, amended claims 1-14 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14 are rejected under 35 U.S.C. 103 as being unpatentable over BOUSSAID (WO 2020064323 A1), in view of GUO (CN 113689431 A), and further in view of BADANES (US 20210209418 A1)
Re Claim 1, BOUSSAID discloses a method for detecting defect for an aeronautical part from at least one image to be tested of the aeronautical part subdivided into a plurality of sub-images (see BOUSSAID : e. g., -- for each pixel of the source image (IMG1), at least one contour characteristic; - a step of determining a contour probability by means of a logistic regression function; - a step of determining a contour (CONT); - a step of readjusting a computer model (MI) of said aerospace part with respect to the contour (CONT) detected in the source image in order to obtain a readjusted computer model (MIc); - a step of generating a simulated image (IMG2) on the basis of the readjusted computer model (MIc); and - a step of comparing the pixels of the simulated image (IMG2) with the source image (IMG1) so as to determine defects in the aerospace part.--, in abstract, and, -- Preferably, the step of calculating the contour characteristic uses at least one histogram operator applied to a parameter of the pixels belonging to a portion of the source image, said portion being centered on said pixel to be characterized, having a scale o and an orientation Q. A histogram operator makes it possible to determine a very relevant contour characteristic by analysis of a portion of image centered on a pixel.--, in page 3 of English version of WO 2020064323 A1 provided as NPL with this Office Action; and,
-- By way of example, the calculation of a contour characteristic * will be presented on a portion of the source image IMG1, also called a “patch”, of scale o and which is centered on a pixel of the source image. IMG1 with coordinates (x, y) and Q orientation. In this example, two adjacent regions of the patch are compared. The adjacent regions are determined based on the size o and Q orientation data. In this example, an image portion is determined by pixel and centered on said pixel taking into account the size o and Q orientation data. This image portion is divided into two adjacent regions defined according to the Q orientation.--, {herein regions, or portions of the source image IMG1 read on claimed limitation of “a plurality of sub-images}); in page 5 of English version of WO 2020064323 A1 provided as NPL with this Office Action);
BOUSSAID however does not explicitly disclose a tested image is subdivided into a plurality of sub-images;
GUO discloses a tested image is subdivided into a plurality of sub-images (see GUO: e.g., --industrial product appearance defect detection and bad detection effect, providing an industrial product appearance defect detection method and device, the method comprises: dividing the image of the industrial product to be detected into multiple sub-regio; combining the sub-regions according to the similarity between the adjacent sub-regions to obtain a plurality of sub-images to form a sub-image set--, in abstract, in English version of CN 113689431 A provided as NPL with this Office Action);
BOUSSAID and GUO are combinable as they are in the same field of endeavor: defects detection based on image processing. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify BOUSSAID ’s method using GUO’s teachings by including a tested image is subdivided into a plurality of sub-images to BOUSSAID ’s dividing a source image into a plurality of portions and regions in order to determine/classification probability that at least one pixel or image region/sub-images belongs to indicate image regions that contains certain properties {“dividing the sample set in the sample set into defect type and good product type, so as to judge whether the to-be-detected industrial product image has appearance defect”} (see GUO: e.g., in abstract); and
the method comprising:
supervisedly training, during a plurality of training epochs, an artificial neural network configured to provide a defect probability for each pixel of a training sub-image of a training image (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action);
for each training epoch and for each image of a validation set, creating a defect probability matrix from merging defect probabilities determined for each pixel of each sub-image of the image of the validation set (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., --in the step b, if the similarity between two adjacent sub-regions is the highest, then the two adjacent sub-regions are combined. repeating the calculation of the similarity between the adjacent sub-regions and the merging of the adjacent sub-regions, until the integrated image of the industrial product to be detected is combined.--, in 2nd para. of page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- after calculating to obtain all the Gaussian similarity, it can obtain a similar matrix S of n * n. then constructing adjacency matrix W, then constructing degree matrix D according to adjacency matrix. Next, calculating the random walk normalization Laplace matrix Lr=I-D-1W, wherein, I represents the unit matrix. compared with Laplace matrix or normalized Laplace matrix, randomly walking the normalized Laplace matrix can make the clustering randomly walk in the same cluster, and almost not to walk to other clusters, so the effect is better. after obtaining the travel return-normalized Laplace matrix, calculating the characteristic value of the matrix,
and calculating the minimum k1 characteristic value corresponding to the characteristic vector, wherein k1 is the preset value. The reason for taking k1 characteristic value and characteristic vector is to reduce the dimension. after the k1 characteristic vector is standardized, obtaining the characteristic matrix T of n * k1.
taking each row of the T as a k1-dimensional sample, then using K-means + + algorithm for clustering. Because only considering all sample clustering is the appearance defect and there is no appearance defect two categories, namely the defect class and good class, therefore setting the K-means + + algorithm to all sample cluster is 2 class. so as to obtain the unsupervised learning model for dividing the sample set into defect type and good product type, and according to whether there is defect type, or whether the number of the sample in the defect type exceeds the preset threshold value to judge whether the to-be-detected industrial product image has appearance defect.--, in page 6/14 of English version of CN 113689431 A provided as NPL with this Office Action);
for each training epoch, determining, from the defect probability matrices associated with said training epoch, a defect detection curve (see BOUSSAID : e. g., -- a step of determining a probability of contour, for each pixel of the source image, by a logistic regression function from the set of contour characteristics measured locally at this pixel, a step of determining a contour in the source image from the determined contour probabilities,
a step of registration of a computer modeling of said aeronautical part with respect to the detected contour of the source image in order to obtain a modeling recalibrated computer, the detected contour fulfilling a similarity criterion function during the registration,
{herein “a logistic regression function” or, “a similarity criterion function” read on claimed “a defect detection curve”, since “the adjusted computer modeling” also reads on claimed “a defect detection curve” as implemented as “adjusted performance of defect detection”} in page 3 of English version of WO 2020064323 A1 provided as NPL with this Office Action);
BOUSSAID as modified by GUO however still do not explicitly disclose a defect detection curve as a function of false alarms;
BADANES discloses a defect detection curve as a function of false alarms (see BADANES: e.g., Fig. 4, and, -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.
[0072] Capture rate (also referred to herein as recall) refers to the ratio of the captured defects over the whole defect population. False alarm rate refers to the ratio of the declared nuisance over the total declared defects. Precision refers to the ratio of the number of DOI over the total declared defects. It is noted that the terms capture rate, recall, false alarm rate, precision etc. are known terminologies in the art and should be understood in its broadest interpretation.--, in [0071]-[0072], and, -- [0082] According to certain embodiments, after processing an image, the supervised model 303 and unsupervised model 304 output the first grade map and the second grade map respectively. A respective threshold (denoted as Th.sub.s to be used for the supervised model, and Th.sub.us to be used for the unsupervised model, as illustrated in FIG. 4) can be applied to each grade map in order to generate a defect detection output, e.g., a defect map (denoted as Output.sub.s for the supervised model, and Output.sub.us for the unsupervised model, as illustrated in FIG. 4). The thresholds can be initially selected during the separate training phase as described above with reference to FIG. 3. According to certain embodiments of the present disclosure, the initially selected threshold of each (or at least one) model can be tuned so as to optimize the capture rate and false alarm rate of the model. In this case, the optimized parameters used for the combination, as mentioned above, are the optimized threshold(s), which can be applied to the corresponding grade map to obtain an optimized defect map. The optimized defect maps of both models can be combined to generate an overall defect detection result 405.
[0083] Specifically, in such cases, the validation set can be used to tune the threshold of one or both of the models according to user-defined capture rate and/or false alarm rate 406. By way of example, a validation image can be processed by both models. If an expected capture rate is pre-defined by the user (e.g., 90%), the thresholds of both models can be increased so as to minimize the false alarm rates, as long as the user-defined capture rate is not compromised. On the other hand, if an expected false alarm rate is pre-defined by the user, the thresholds of both models can be decreased so as to increase the capture rate, as long as the user-defined false rate is not compromised. A graph 408 illustrating the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) is shown in FIG. 4. As illustrated, a high performance model would have a high true positive rate and a low false positive rate, which would be within the top left target area. This means that most predictions of the model are consistent with the ground truth, and most of the declared defects are detected.--, in [0082]-[0083]);
BOUSSAID (as modified by GUO) and BADANES are combinable as they are in the same field of endeavor: defects detection based on machine learning and image processing. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify BOUSSAID (as modified by GUO)’s method using BADANES’s teachings by including a defect detection curve as a function of false alarms to BOUSSAID (as modified by GUO) ’s model evaluation in order to validate the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) (see BADANES: e.g., in [0071]-[0072], and [0082]-[0083]);
BOUSSAID as modified by GUO and BADANES further disclose determining an ensemble of the N best training epochs of the plurality of training epochs as a function of the defect detection curves as a function of false alarms and a criterion in the field, N being an integer greater than or equal to 1 (see BADANES: e.g., -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]);
for each training epoch of the N best epochs, applying, to each sub-image of the image to be tested, the artificial neural network with parameters associated with said epoch, in order to obtain a tested sub-image associated with said training epoch (see BADANES: e.g., -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]), and
for each sub-image of the image to be tested, merging the result matrices of the N tested sub-images to obtain probabilities of the presence of defect on the sub-image (see BADANES: e.g., -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]; also see GUO: e.g., --in the step b, if the similarity between two adjacent sub-regions is the highest, then the two adjacent sub-regions are combined. repeating the calculation of the similarity between the adjacent sub-regions and the merging of the adjacent sub-regions, until the integrated image of the industrial product to be detected is combined.--, in 2nd para. of page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- after calculating to obtain all the Gaussian similarity, it can obtain a similar matrix S of n * n. then constructing adjacency matrix W, then constructing degree matrix D according to adjacency matrix. Next, calculating the random walk normalization Laplace matrix Lr=I-D-1W, wherein, I represents the unit matrix. compared with Laplace matrix or normalized Laplace matrix, randomly walking the normalized Laplace matrix can make the clustering randomly walk in the same cluster, and almost not to walk to other clusters, so the effect is better. after obtaining the travel return-normalized Laplace matrix, calculating the characteristic value of the matrix,
and calculating the minimum k1 characteristic value corresponding to the characteristic vector, wherein k1 is the preset value. The reason for taking k1 characteristic value and characteristic vector is to reduce the dimension. after the k1 characteristic vector is standardized, obtaining the characteristic matrix T of n * k1.
taking each row of the T as a k1-dimensional sample, then using K-means + + algorithm for clustering. Because only considering all sample clustering is the appearance defect and there is no appearance defect two categories, namely the defect class and good class, therefore setting the K-means + + algorithm to all sample cluster is 2 class. so as to obtain the unsupervised learning model for dividing the sample set into defect type and good product type, and according to whether there is defect type, or whether the number of the sample in the defect type exceeds the preset threshold value to judge whether the to-be-detected industrial product image has appearance defect.--, in page 6/14 of English version of CN 113689431 A provided as NPL with this Office Action).
Re Claim 2, BOUSSAID as modified by GUO and BADANES further disclose wherein the supervised training is carried out, for the plurality of epochs, from a training database, the number of training epochs of the plurality of training epochs being determined by a stop criterion, each training epoch of the plurality of training epochs being associated with a number, the training database comprising a plurality of sub-images for each image of a plurality of training images of aeronautical parts, each sub-image being associated with defect information for each pixel of the sub-image (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
.
Re Claim 3, BOUSSAID as modified by GUO and BADANES further disclose wherein the step of creating a defect probability matrix associated with each training epoch and associated with each image of the validation set is carried out according to the following sub-steps of:
for each sub-image of the image of the validation set, using, on the sub-image, the artificial neural network with the parameters associated with said epoch, to obtain a defect probability for each pixel of the sub-image (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]),
associating with each pixel of the image of the validation set, a pixel defect probability calculated from merging each defect probability obtained for the pixel (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Re Claim 4, BOUSSAID as modified by GUO and BADANES further disclose wherein the step of determining a defect detection curve as a function of false alarms associated with a training epoch is carried out according to the following sub-steps of:
for each threshold of a plurality of thresholds included in a predefined threshold interval, and for each image of the validation set:
thresholding the defect probability matrix associated with said image and with said epoch by the threshold (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]);
for each ensemble of adjacent pixels of said image of the validation set comprising only pixels associated with a non-zero pixel defect probability in the thresholded defect probability matrix associated with the image (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]):
if no pixel of the ensemble of pixels is associated with non- zero defect information in the validation set, incrementing a number of false alarms associated with the threshold (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]);
for each ensemble of adjacent pixels with non-zero defect information in the validation set, if at least one pixel of this the ensemble of pixels is associated with a non-zero pixel defect probability in the thresholded defect probability matrix, incrementing a number of detections associated with the threshold (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]);
Plotting plotting a curve, associated with said epoch, representing the number of detections as a function of the number of false alarms, including one point per threshold, corresponding to the number of detections and to the number of false alarms associated with the threshold (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Re Claim 5, BOUSSAID as modified by GUO and BADANES further disclose for each training epoch from the ensemble of N best training epochs, the result of testing a sub-image is a matrix, each coefficient of which corresponds to a defect probability for each pixel of the sub-image (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., --in the step b, if the similarity between two adjacent sub-regions is the highest, then the two adjacent sub-regions are combined. repeating the calculation of the similarity between the adjacent sub-regions and the merging of the adjacent sub-regions, until the integrated image of the industrial product to be detected is combined.--, in 2nd para. of page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- after calculating to obtain all the Gaussian similarity, it can obtain a similar matrix S of n * n. then constructing adjacency matrix W, then constructing degree matrix D according to adjacency matrix. Next, calculating the random walk normalization Laplace matrix Lr=I-D-1W, wherein, I represents the unit matrix. compared with Laplace matrix or normalized Laplace matrix, randomly walking the normalized Laplace matrix can make the clustering randomly walk in the same cluster, and almost not to walk to other clusters, so the effect is better. after obtaining the travel return-normalized Laplace matrix, calculating the characteristic value of the matrix,
and calculating the minimum k1 characteristic value corresponding to the characteristic vector, wherein k1 is the preset value. The reason for taking k1 characteristic value and characteristic vector is to reduce the dimension. after the k1 characteristic vector is standardized, obtaining the characteristic matrix T of n * k1.
taking each row of the T as a k1-dimensional sample, then using K-means + + algorithm for clustering. Because only considering all sample clustering is the appearance defect and there is no appearance defect two categories, namely the defect class and good class, therefore setting the K-means + + algorithm to all sample cluster is 2 class. so as to obtain the unsupervised learning model for dividing the sample set into defect type and good product type, and according to whether there is defect type, or whether the number of the sample in the defect type exceeds the preset threshold value to judge whether the to-be-detected industrial product image has appearance defect.--, in page 6/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]).
Re Claim 6, BOUSSAID as modified by GUO and BADANES further disclose wherein the artificial neural network is configured to provide, from a sub-image, a defect probability per pixel of the sub-image, each sub-image of the training database and each sub-image of the validation database being associated with defect information per pixel of the sub-image (see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; see BADANES: e.g., Fig. 4, and, -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.
[0072] Capture rate (also referred to herein as recall) refers to the ratio of the captured defects over the whole defect population. False alarm rate refers to the ratio of the declared nuisance over the total declared defects. Precision refers to the ratio of the number of DOI over the total declared defects. It is noted that the terms capture rate, recall, false alarm rate, precision etc. are known terminologies in the art and should be understood in its broadest interpretation.--, in [0071]-[0072], and, -- [0082] According to certain embodiments, after processing an image, the supervised model 303 and unsupervised model 304 output the first grade map and the second grade map respectively. A respective threshold (denoted as Th.sub.s to be used for the supervised model, and Th.sub.us to be used for the unsupervised model, as illustrated in FIG. 4) can be applied to each grade map in order to generate a defect detection output, e.g., a defect map (denoted as Output.sub.s for the supervised model, and Output.sub.us for the unsupervised model, as illustrated in FIG. 4). The thresholds can be initially selected during the separate training phase as described above with reference to FIG. 3. According to certain embodiments of the present disclosure, the initially selected threshold of each (or at least one) model can be tuned so as to optimize the capture rate and false alarm rate of the model. In this case, the optimized parameters used for the combination, as mentioned above, are the optimized threshold(s), which can be applied to the corresponding grade map to obtain an optimized defect map. The optimized defect maps of both models can be combined to generate an overall defect detection result 405.
[0083] Specifically, in such cases, the validation set can be used to tune the threshold of one or both of the models according to user-defined capture rate and/or false alarm rate 406. By way of example, a validation image can be processed by both models. If an expected capture rate is pre-defined by the user (e.g., 90%), the thresholds of both models can be increased so as to minimize the false alarm rates, as long as the user-defined capture rate is not compromised. On the other hand, if an expected false alarm rate is pre-defined by the user, the thresholds of both models can be decreased so as to increase the capture rate, as long as the user-defined false rate is not compromised. A graph 408 illustrating the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) is shown in FIG. 4. As illustrated, a high performance model would have a high true positive rate and a low false positive rate, which would be within the top left target area. This means that most predictions of the model are consistent with the ground truth, and most of the declared defects are detected.--, in [0082]-[0083]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Re Claim 7, BOUSSAID as modified by GUO and BADANES further disclose wherein the artificial neural network is configured so as to provide, from a sub-image, a defect probability for the ensemble of pixels of the sub-image, each sub-image of the training database and each sub-image of the validation database being associated with defect information for the ensemble of pixels of the sub-image (see GUO: e.g., -- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; see BADANES: e.g., Fig. 4, and, -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.
[0072] Capture rate (also referred to herein as recall) refers to the ratio of the captured defects over the whole defect population. False alarm rate refers to the ratio of the declared nuisance over the total declared defects. Precision refers to the ratio of the number of DOI over the total declared defects. It is noted that the terms capture rate, recall, false alarm rate, precision etc. are known terminologies in the art and should be understood in its broadest interpretation.--, in [0071]-[0072], and, -- [0082] According to certain embodiments, after processing an image, the supervised model 303 and unsupervised model 304 output the first grade map and the second grade map respectively. A respective threshold (denoted as Th.sub.s to be used for the supervised model, and Th.sub.us to be used for the unsupervised model, as illustrated in FIG. 4) can be applied to each grade map in order to generate a defect detection output, e.g., a defect map (denoted as Output.sub.s for the supervised model, and Output.sub.us for the unsupervised model, as illustrated in FIG. 4). The thresholds can be initially selected during the separate training phase as described above with reference to FIG. 3. According to certain embodiments of the present disclosure, the initially selected threshold of each (or at least one) model can be tuned so as to optimize the capture rate and false alarm rate of the model. In this case, the optimized parameters used for the combination, as mentioned above, are the optimized threshold(s), which can be applied to the corresponding grade map to obtain an optimized defect map. The optimized defect maps of both models can be combined to generate an overall defect detection result 405.
[0083] Specifically, in such cases, the validation set can be used to tune the threshold of one or both of the models according to user-defined capture rate and/or false alarm rate 406. By way of example, a validation image can be processed by both models. If an expected capture rate is pre-defined by the user (e.g., 90%), the thresholds of both models can be increased so as to minimize the false alarm rates, as long as the user-defined capture rate is not compromised. On the other hand, if an expected false alarm rate is pre-defined by the user, the thresholds of both models can be decreased so as to increase the capture rate, as long as the user-defined false rate is not compromised. A graph 408 illustrating the performance of the models with reference to true positive rate (i.e., capture rate) and false positive rate (i.e., false alarm rate) is shown in FIG. 4. As illustrated, a high performance model would have a high true positive rate and a low false positive rate, which would be within the top left target area. This means that most predictions of the model are consistent with the ground truth, and most of the declared defects are detected.--, in [0082]-[0083]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Re Claim 8, BOUSSAID as modified by GUO and BADANES further disclosewherein the step of determining the defect detection curve further comprises a step of correcting by deleting each point corresponding to a threshold for which the number of false alarms associated with the threshold is less than the number of false alarms associated with the immediately following threshold in the interval of predefined thresholds following the plotting step, the deleting step being repeated for each deleted point starting from the lowest remaining threshold (see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Re Claim 9, BOUSSAID as modified by GUO and BADANES further disclose wherein the corrected curve (1081) is normalised by adding an initial point associated with a zero number of detections and a zero number of false alarms and adding a final point associated with the maximum number of detections of the corrected curve and with a predefined maximum number of false alarms, following the correction step (see BADANES: e.g., Fig. 4, and, --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Re Claim 10, BOUSSAID as modified by GUO and BADANES further disclose wherein the criterion in the field for determining an ensemble of the N best training epochs of the plurality of training epochs as a function of the defect detection curves is the area under the curve (see BADANES: e.g., Fig. 4, and, --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]; and,
-- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Re Claim 11, BOUSSAID as modified by GUO and BADANES further disclose wherein each pixel of an image is associated with a position in the image and the step of creating the defect probability matrix for an image is further performed from the position associated with each pixel of the image (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., --in the step b, if the similarity between two adjacent sub-regions is the highest, then the two adjacent sub-regions are combined. repeating the calculation of the similarity between the adjacent sub-regions and the merging of the adjacent sub-regions, until the integrated image of the industrial product to be detected is combined.--, in 2nd para. of page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- after calculating to obtain all the Gaussian similarity, it can obtain a similar matrix S of n * n. then constructing adjacency matrix W, then constructing degree matrix D according to adjacency matrix. Next, calculating the random walk normalization Laplace matrix Lr=I-D-1W, wherein, I represents the unit matrix. compared with Laplace matrix or normalized Laplace matrix, randomly walking the normalized Laplace matrix can make the clustering randomly walk in the same cluster, and almost not to walk to other clusters, so the effect is better. after obtaining the travel return-normalized Laplace matrix, calculating the characteristic value of the matrix,
and calculating the minimum k1 characteristic value corresponding to the characteristic vector, wherein k1 is the preset value. The reason for taking k1 characteristic value and characteristic vector is to reduce the dimension. after the k1 characteristic vector is standardized, obtaining the characteristic matrix T of n * k1.
taking each row of the T as a k1-dimensional sample, then using K-means + + algorithm for clustering. Because only considering all sample clustering is the appearance defect and there is no appearance defect two categories, namely the defect class and good class, therefore setting the K-means + + algorithm to all sample cluster is 2 class. so as to obtain the unsupervised learning model for dividing the sample set into defect type and good product type, and according to whether there is defect type, or whether the number of the sample in the defect type exceeds the preset threshold value to judge whether the to-be-detected industrial product image has appearance defect.--, in page 6/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]).
Re Claim 12, BOUSSAID as modified by GUO and BADANES further disclose wherein upon creating the defect probability matrix for an image, merging is performed, for each pixel of the image associated with a plurality of defect probabilities, of the plurality of defect probabilities associated with the pixel (see BOUSSAID : e. g., --Preferably, the training images are obtained by varying the projection angles in a 2D plane and by varying the acquisition parameters by imagery. Thus, we obtain a large number of images corresponding to different types of acquisition so as to form a relevant basis of comparison when learning because of its richness.
Preferably, the verified contour of a learning image is obtained by projection of a silhouette of the computer modeling of the aeronautical part on the learning image. The outline of the learning image is verified and forms a ground truth since it is obtained directly from a computer modeling whose relevance is recognized.
According to a preferred aspect of the invention, each contour characteristic of a pixel depending on an orientation parameter Q, the step of determining a probability of contour for each pixel of the source image implements a Weakly supervised learning step of the “Multiple Instance Learning” type in order to allow learning without knowledge of said orientation parameter Q. Such learning allows a global classification, by bags, in order to take into account the latent orientation parameter Q .--, from the second last para. in page 3 to 1st para. of page 4 of English version of WO 2020064323 A1 provided as NPL with this Office Action; also see GUO: e.g., --in the step b, if the similarity between two adjacent sub-regions is the highest, then the two adjacent sub-regions are combined. repeating the calculation of the similarity between the adjacent sub-regions and the merging of the adjacent sub-regions, until the integrated image of the industrial product to be detected is combined.--, in 2nd para. of page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- after calculating to obtain all the Gaussian similarity, it can obtain a similar matrix S of n * n. then constructing adjacency matrix W, then constructing degree matrix D according to adjacency matrix. Next, calculating the random walk normalization Laplace matrix Lr=I-D-1W, wherein, I represents the unit matrix. compared with Laplace matrix or normalized Laplace matrix, randomly walking the normalized Laplace matrix can make the clustering randomly walk in the same cluster, and almost not to walk to other clusters, so the effect is better. after obtaining the travel return-normalized Laplace matrix, calculating the characteristic value of the matrix,
and calculating the minimum k1 characteristic value corresponding to the characteristic vector, wherein k1 is the preset value. The reason for taking k1 characteristic value and characteristic vector is to reduce the dimension. after the k1 characteristic vector is standardized, obtaining the characteristic matrix T of n * k1.
taking each row of the T as a k1-dimensional sample, then using K-means + + algorithm for clustering. Because only considering all sample clustering is the appearance defect and there is no appearance defect two categories, namely the defect class and good class, therefore setting the K-means + + algorithm to all sample cluster is 2 class. so as to obtain the unsupervised learning model for dividing the sample set into defect type and good product type, and according to whether there is defect type, or whether the number of the sample in the defect type exceeds the preset threshold value to judge whether the to-be-detected industrial product image has appearance defect.--, in page 6/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., --[0065] The runtime image can be processed (204) using a supervised model component (e.g., by the supervised model 104) to obtain a first output (also referred to herein as supervised output) indicative of the estimated presence of first defects on the runtime image. In some embodiments, the first output is a first grade map representative of estimated probabilities of the first defects on the runtime image (e.g., a probability map). In some cases, a threshold can be applied to the pixel values of the first grade map, giving rise to a first defect map indicative of the estimated presence of defects (e.g., a binary map indicative of the presence of defects or not). By way of example, the first grade map can include pixel values in the range of [0, 1], and a threshold of 0.6 can be applied to each pixel value in the grade map. Any pixel value that is larger than the threshold 0.6 can be marked as a defect with a value of 1 in the corresponding defect map. The supervised model component can be trained in a setup stage using a first training set including at least a plurality of first images each representative of at least a portion of the specimen and label data indicative of first defect distribution on the first images. The label data can be obtained from a review tool or from manual classification, and can include the location of defects on the first images and the class of defects.
[0066] Separately, the runtime image can be processed (206) using an unsupervised model component (e.g., by the unsupervised model 108) to obtain a second output (also referred to herein as unsupervised output) indicative of the estimated presence of second defects on the runtime image. In some embodiments, the second output is a second grade map representative of estimated probabilities of the second defects on the runtime image. In some cases, a threshold can be applied to the pixel values of the second grade map, giving rise to a second defect map indicative of the estimated presence of defects. The unsupervised network is trained using a second training set including a plurality of second images of the specimen, each second image being a reference image of a first image.--, in [0065]-[0066]).
Re Claim 13, BOUSSAID as modified by GUO and BADANES further disclose wherein the defect probabilities for each sub-image of the image to be tested obtained from the merging of the result matrices of the N tested sub-images are merged in order to obtain defect probabilities for the image to be tested (see BADANES: e.g., -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]; also see GUO: e.g., --in the step b, if the similarity between two adjacent sub-regions is the highest, then the two adjacent sub-regions are combined. repeating the calculation of the similarity between the adjacent sub-regions and the merging of the adjacent sub-regions, until the integrated image of the industrial product to be detected is combined.--, in 2nd para. of page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- after calculating to obtain all the Gaussian similarity, it can obtain a similar matrix S of n * n. then constructing adjacency matrix W, then constructing degree matrix D according to adjacency matrix. Next, calculating the random walk normalization Laplace matrix Lr=I-D-1W, wherein, I represents the unit matrix. compared with Laplace matrix or normalized Laplace matrix, randomly walking the normalized Laplace matrix can make the clustering randomly walk in the same cluster, and almost not to walk to other clusters, so the effect is better. after obtaining the travel return-normalized Laplace matrix, calculating the characteristic value of the matrix,
and calculating the minimum k1 characteristic value corresponding to the characteristic vector, wherein k1 is the preset value. The reason for taking k1 characteristic value and characteristic vector is to reduce the dimension. after the k1 characteristic vector is standardized, obtaining the characteristic matrix T of n * k1.
taking each row of the T as a k1-dimensional sample, then using K-means + + algorithm for clustering. Because only considering all sample clustering is the appearance defect and there is no appearance defect two categories, namely the defect class and good class, therefore setting the K-means + + algorithm to all sample cluster is 2 class. so as to obtain the unsupervised learning model for dividing the sample set into defect type and good product type, and according to whether there is defect type, or whether the number of the sample in the defect type exceeds the preset threshold value to judge whether the to-be-detected industrial product image has appearance defect.--, in page 6/14 of English version of CN 113689431 A provided as NPL with this Office Action).
Re Claim 14, BOUSSAID as modified by GUO and BADANES further disclose wherein N is an integer greater than or equal to 2, and wherein merging the result matrices of the N tested sub-images comprises calculating, for each position, an average of the coefficients occupying said position in each of the N result matrices see GUO: e.g., --in the step b, if the similarity between two adjacent sub-regions is the highest, then the two adjacent sub-regions are combined. repeating the calculation of the similarity between the adjacent sub-regions and the merging of the adjacent sub-regions, until the integrated image of the industrial product to be detected is combined.--, in 2nd para. of page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; and,
-- the sub-graph obtained by the combining module 20 can include a plurality of sub-images with appearance defects and no sub-graph of appearance defects;…. the plurality of sub-graph in the sub-graph set is divided into jumbo size, middle size, small size three types by the K-means + + algorithm, the sub-graph in the jumbo size is adjusted to the jumbo size long-width value, the sub-graph in the size class is adjusted to the size corresponding to the long-width value; The sub-picture in the small size class is adjusted to be the long-width value corresponding to the small size.
the forming module 50 the sub-graph of the K respectively in turn input the convolutional neural network module one to perform feature extraction, the convolutional neural network module is VGG, Resnet, Vovnet, Efficientnet, Hrnet, Bifpn, Mobilenet; Shufflenet and so on; the weight of the convolutional neural network module can be obtained by classifying and learning the image with the appearance defect and the image without appearance defect in the past; that is to say, there is a pre-training model based on the appearance defect;--, in --, in page 5/14 of English version of CN 113689431 A provided as NPL with this Office Action; also see BADANES: e.g., -- a set of thresholds can be applied on the first grade map to generate corresponding defect maps. False alarm rate (FAR) and/or capture rate (CR) can be calculated based on the prediction results (e.g., the defect maps) and compared with an initial FAR as obtained from the labeled data or from the user for assessing the prediction performance of the supervised model. A threshold with the best FAR and/or CR can be selected to be used in subsequent training or in runtime.--, in [0071]; and, -- [0092] As illustrated, a segmentation model 602 is operatively connected to the supervised model 303 and the unsupervised model 304. The segmentation model 602 can be trained using a third training set 401 (also referred to as a validation set) based on the outputs of the supervised model 303 and unsupervised model 304. As aforementioned, the validation set includes a set of validation images and corresponding label data. Specifically, a validation image can be fed into the trained supervised and unsupervised models, and the outputs of both models can be provided as input to the segmentation model 602. The segmentation model learns to combine the outputs of both models and generates a combined/composite grade map, which, after thresholding, can be compared with ground truth data (e.g., the labeled data corresponding to the validation image). The parameters in the segmentation model 602 can be optimized through minimizing the difference between the prediction and the ground truth. The threshold to be applied on the composite grade map can also be tuned during training so as to get an optimized capture rate and/or false alarm rate. In such a way, the segmentation model can be trained to learn how to best combine the outputs of both models, and provide an optimized defect detection result 604.--, in [0092]-[0095]).
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights