DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) were submitted on 9/22/2024 and 4/14/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Objections
Claims 2-4, 8, 12, 13, and 17 are objected to because of the following informalities:
Claim 2, “image restored model” lacks antecedent basis. Examiner suggests amending to “image restoration model”.
Claims 3, 4, 12, and 13, elements are listed in the alternative (“at least one of”) but use “and” as a conjunction. Examiner suggests amending to “or” for clarity.
Claim 8 and 17, “the product to be inspected” lacks antecedent basis. Examiner suggests amending to “a product to be inspected”.
Appropriate correction is required.
Positive Statement Regarding - 35 USC § 101
The Examiner’s 35 U.S.C. 101 analysis recognizes that the claimed subject matter is directed to a practical application of a technical solution. The claimed elements, taken as a whole, recite specific steps directed at implementing a machine learning method for defect detection. Because the claims recite specific, claimed steps and structural elements that produce a tangible technical result, they are not directed to an abstract idea absent additional inventive concept limitations. Accordingly, the record supports a positive 101 determination for the present claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 8, 10-13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (Kim, Daehwan, et al. "Spatial contrastive learning for anomaly detection and localization." IEEE Access 10 (2022): 17366-17376) (hereafter, “Kim”) (IDS) in view of Chen et al. (US 2021/0105578) (hereafter, “Chen”).
Regarding claim 1, Kim discloses an artificial intelligence device, comprising: [a memory configured to store a first normal product image; a learning processor] configured to train an image restoration model through inputting the first normal product image into the image restoration model as learning data to output a normal restored image similar to the first normal product image (Page 17369, Algorithm 1; Page 17369, left column, paragraph 2-3, augmentations such as contrast, brightness, hue, 90_ rotation, and resize … The autoencoder is trained to generate Iinput with the generated image Imcutp as input. Examiner considers the autoencoder as the restoration model. The input minibatch in algorithm 1 is considered a set of “first normal product images”. Since the autoencoder is trained to reconstruct images based on augmented input images, the output is considered “similar to the first normal product image”); and [a processor] configured to: i) modify the first normal product image to generate a first normal modified image belonging to a normal classification (Page 17369, Algorithm 1, line 2, Iin = augment(xk); Page 17369, left column, paragraph 2, random augmentations such as contrast, brightness, hue, 90_ rotation, and resize are applied. Examiner considers the input image xk and the augmented image Iin to both be “normal classification” since no defect is introduced), and [increase a number of a second normal product image belonging to the normal classification, the second normal product image including the first normal product image and the first normal modified image]; ii) modify at least one of the second normal product image belonging to the normal classification to generate an abnormal modified image belonging to an abnormal classification (Page 17369, Algorithm 1, line 3; Page 17369, left column, paragraph 3, this method assumes the
M-CutPaste area as noise and helps learn denoising); and iii) input the abnormal modified image into the image restoration model to acquire an abnormal restored image output from the image restoration model (Fig. 4; Page 17369, left column, paragraph 3, The autoencoder is trained to generate Iinput with the generated image Imcutp as input. Examiner considers Imcutp as the abnormal modified image and the output of the autoencoder, illustrated by the reconstruction image in Fig. 4, as the abnormal restored image), wherein the processor is further configured to: a) acquire an inspection product image for a product to be inspected (Page 17370, right column, paragraph 5, The remaining 20% of the image is used to check the training loss. Examiner considers the images in the testing set as the inspection product image); b) input the inspection product image into the image restoration model (Fig. 5. Fig. 5 illustrates inputting images into the autoencoder which is being considered as the “restoration model”), and acquire a restored inspection product image output from the image restoration model (Fig. 5; Page 17370, right column, paragraph 2, the patch set of the image restored through the autoencoder); and c) determine whether the product to be inspected is defective (Page 17368, left column, last paragraph, we present the anomaly detection algorithm; Page 17370, right column, paragraph 3, The anomaly score is de_ned as the maximum value of the final anomaly map) by using a distance between a first expression vector of the inspection product image and a second expression vector of the restored inspection product image (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D 1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Examiner considers the two embedding vectors sets fcon(pi) and fcon(qj) as the first and second “expression vectors”. Eqn. 7 calculates the distance between the two embedding vectors).
However, Kim fails to explicitly disclose a memory, a learning processor, a processor, and increase a number of a second normal product image belonging to the normal classification, the second normal product image including the first normal product image and the first normal modified image.
Chen teaches a memory (¶0235, a flash memory), a learning processor (¶0008, reconstruction layers including at least one third processor; ¶0237, the DNN or CNN model may be manufactured in a form of a dedicated hardware chip for AI), a processor (¶0008, a data augmenter including at least one first processor; ¶0237, the DNN or CNN model may be manufactured in a form of a dedicated hardware chip for AI), and increase a number of a second normal product image belonging to the normal classification, the second normal product image including the first normal product image and the first normal modified image (¶0144, Data augmentation aims to enhance the diversity and sample size of the training data; ¶0146, the training data set is enlarged. Examiner is interpreting enhancing and enlarging the training data set as combining the original data with the augmented data. Further, Examiner is interpreting the combined group of original and augmented data to be “second normal product image” as described in ¶0026 of the instant application).
Both Kim and Chen are analogous to the claimed invention because Kim is directed towards anomaly detection methods and Chen is directed towards models with reconstruction networks. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the multiple processors and data augmentation of Chen into the model of Kim. The suggestion/motivation for doing so would have been to improve performance, as suggested by Chen at ¶0044, increases average F1 score by 17.8% and improves the worst-case accuracy by 20.2%.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen to obtain the invention as specified in claim 1.
Regarding claim 2, in which claim 1 is incorporated, Kim discloses wherein the learning processor is configured to train the image restored model so that a mean square error (MSE) of a pixel value of the first normal product image and a pixel value of the first normal restored image is minimized (Eqn. 1; Algorithm 1, line 5, update networks fAE to minimize L. Eqn. 1 calculates the MSE).
Regarding claim 3, in which claim 1 is incorporated, Kim discloses wherein the processor is configured to apply at least one of brightness change, color change, contrast change, rotation, and rescale to the first normal product image to generate the second normal product image belonging to the normal classification (Page 17369, left column, paragraph 2, augmentations such as contrast, brightness, hue, 90_ rotation, and resize. Since the limitation is recited in the alternative, Examiner considers this citation to fully disclose the limitation) and [increase the number of the second normal product image].
However, Kim fails to explicitly disclose increase the number of the second normal product image.
Chen teaches increase the number of the second normal product image (¶0144, Data augmentation aims to enhance the diversity and sample size of the training data; ¶0146, the training data set is enlarged. Examiner is interpreting enhancing and enlarging the training data set as combining the original data with the augmented data).
Both Kim and Chen are analogous to the claimed invention because Kim is directed towards anomaly detection methods and Chen is directed towards models with reconstruction networks. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the multiple processors and data augmentation of Chen into the model of Kim. The suggestion/motivation for doing so would have been to improve performance, as suggested by Chen at ¶0044, increases average F1 score by 17.8% and improves the worst-case accuracy by 20.2%.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen to obtain the invention as specified in claim 3.
Regarding claim 4, in which claim 3 is incorporated, Kim discloses wherein the processor is configured to apply at least one of cut-out, cut-paste, and noise addition to the at least one second normal product image belonging to the normal classification to generate the abnormal modified image belonging to the abnormal classification (Fig. 4; Page 17369, left column, paragraph 2, I’aug is generated by an inversed cutout mask which is a random size white square … a modified cutpaste image Imcutp is made by combining Icutout and I’aug. Examiner considers Imcutp as the abnormal modified image. Since the limitation is recited in the alternative, Examiner considers this citation to fully disclose the limitation).
Regarding claim 8, in which claim 1 is incorporated, Kim discloses wherein the processor is configured to acquire a distance between the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D1; … ;m), and the patch set of the image restored through the
autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Since the distance between the two embedding vectors is used for determining the anomaly map, it must be acquired by the processor at some point), and determine whether the product to be inspected is defective according to the distance between the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m)… the minimum value is defined as one anomaly score by measuring the distance of L2 from all vector sets fcon(pi). Eqn. 7 calculates the distance between the two embedding vectors. Examiner considers the anomaly score as determining whether the product is defective).
Regarding claim 10, Kim discloses a method for detecting a defective product, comprising the steps of: acquiring, by [a processor], a first normal product image (Page 17369, Algorithm 1; The input minibatch in algorithm 1 is considered a set of “first normal product images”); training, by [a learning processor], an image restoration model through inputting the first normal product image into the image restoration model as learning data to output a normal restored image similar to the first normal product image (Page 17369, Algorithm 1; Page 17369, left column, paragraph 2-3, augmentations such as contrast, brightness, hue, 90_ rotation, and resize … The autoencoder is trained to generate Iinput with the generated image Imcutp as input. Examiner considers the autoencoder as the restoration model. Since the autoencoder is trained to reconstruct images based on augmented input images, the output is considered “similar to the first normal product image”); modifying, by [the processor], the first normal product image to generate a first normal modified image belonging to a normal classification (Page 17369, Algorithm 1, line 2, Iin = augment(xk); Page 17369, left column, paragraph 2, random augmentations such as contrast, brightness, hue, 90_ rotation, and resize are applied. Examiner considers the input image xk and the augmented image Iin to both be “normal classification” since no defect is introduced) and [increasing a number of a second normal product image belonging to the normal classification, the second normal product image including the first normal product image and the first normal modified image]; modifying, by the processor, at least one of the second normal product image belonging to the normal classification to generate an abnormal modified image belonging to an abnormal classification (Page 17369, Algorithm 1, line 3; Page 17369, left column, paragraph 3, this method assumes the M-CutPaste area as noise and helps learn denoising); inputting, by the processor, the abnormal modified image into the image restoration model to acquire an abnormal restored image output from the image restoration model (Fig. 4; Page 17369, left column, paragraph 3, The autoencoder is trained to generate Iinput with the generated image Imcutp as input. Examiner considers Imcutp as the abnormal modified image and the output of the autoencoder, illustrated by the reconstruction image in Fig. 4, as the abnormal restored image); acquiring, by the processor, an inspection product image for a product to be inspected (Page 17370, right column, paragraph 5, The remaining 20% of the image is used to check the training loss. Examiner considers the images in the testing set as the inspection product image), inputting, by the processor, the inspection product image into the image restoration model (Fig. 5. Fig. 5 illustrates inputting images into the autoencoder which is being considered as the “restoration model”), and acquiring a restored inspection product image output from the image restoration model (Fig. 5; Page 17370, right column, paragraph 2, the patch set of the image restored through the autoencoder); and determining, by the processor, whether the product to be inspected is defective by using a distance between a first expression vector of the inspection product image and a second expression vector of the restored inspection product image (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D 1; … ;m), and the patch set of the image restored through the
autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Examiner considers the two embedding vectors sets fcon(pi) and fcon(qj) as the first and second “expression vectors”. Eqn. 7 calculates the distance between the two embedding vectors).
However, Kim fails to explicitly disclose a processor, a learning processor, and increasing a number of a second normal product image belonging to the normal classification, the second normal product image including the first normal product image and the first normal modified image.
Chen teaches a processor (¶0008, a data augmenter including at least one first processor; ¶0237, the DNN or CNN model may be manufactured in a form of a dedicated hardware chip for AI), a learning processor (¶0008, reconstruction layers including at least one third processor; ¶0237, the DNN or CNN model may be manufactured in a form of a dedicated hardware chip for AI), and increasing a number of a second normal product image belonging to the normal classification, the second normal product image including the first normal product image and the first normal modified image (¶0144, Data augmentation aims to enhance the diversity and sample size of the training data; ¶0146, the training data set is enlarged. Examiner is interpreting enhancing and enlarging the training data set as combining the original data with the augmented data. Further, Examiner is interpreting the combined group of original and augmented data to be “second normal product image” as described in ¶0026 of the instant application).
Both Kim and Chen are analogous to the claimed invention because Kim is directed towards anomaly detection methods and Chen is directed towards models with reconstruction networks. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the multiple processors and data augmentation of Chen into the model of Kim. The suggestion/motivation for doing so would have been to improve performance, as suggested by Chen at ¶0044, increases average F1 score by 17.8% and improves the worst-case accuracy by 20.2%.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen to obtain the invention as specified in claim 10.
Regarding claim 11, in which claim 10 is incorporated, Kim discloses wherein the step of training the image restoration model includes a step of training the image restoration model so that a mean square error (MSE) of a pixel value of the first normal product image and a pixel value of the first normal restored image is minimized (Eqn. 1; Algorithm 1, line 5, update networks fAE to minimize L. Eqn. 1 calculates the MSE).
Regarding claim 12, in which claim 10 is incorporated, Kim discloses wherein the step of modifying the first normal product image includes a step of applying at least one of brightness change, color change, contrast change, rotation, and rescale to the first normal product image to generate the second normal product image belonging to the normal classification (Page 17369, left column, paragraph 2, augmentations such as contrast, brightness, hue, 90_ rotation, and resize. Since the limitation is recited in the alternative, Examiner considers this citation to fully disclose the limitation) and [increase the number of the second normal product image].
However, Kim fails to explicitly disclose increase the number of the second normal product image.
Chen teaches increase the number of the second normal product image (¶0144, Data augmentation aims to enhance the diversity and sample size of the training data; ¶0146, the training data set is enlarged. Examiner is interpreting enhancing and enlarging the training data set as combining the original data with the augmented data).
Both Kim and Chen are analogous to the claimed invention because Kim is directed towards anomaly detection methods and Chen is directed towards models with reconstruction networks. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the multiple processors and data augmentation of Chen into the model of Kim. The suggestion/motivation for doing so would have been to improve performance, as suggested by Chen at ¶0044, increases average F1 score by 17.8% and improves the worst-case accuracy by 20.2%.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen to obtain the invention as specified in claim 12.
Regarding claim 13, in which claim 12 is incorporated, Kim discloses wherein the step of modifying the second normal product image includes a step of applying at least one of cut-out, cut-paste, and noise addition to the at least one second normal product image belonging to the normal classification to generate the abnormal modified image belonging to the abnormal classification (Fig. 4; Page 17369, left column, paragraph 2, I’aug is generated by an inversed cutout mask which is a random size white square … a modified cutpaste image Imcutp is made by combining Icutout and I’aug. Examiner considers Imcutp as the abnormal modified image).
Regarding claim 17, in which claim 10 is incorporated, Kim discloses wherein the step of determining whether the product to be inspected is defective includes the steps of: acquiring a distance between the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D
1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Since the distance between the two embedding vectors is used for determining the anomaly map, it must be acquired by the processor at some point); and determining whether the product to be inspected is defective according to the distance between the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m)… the minimum value is defined as one anomaly score by measuring the distance of L2 from all vector sets fcon(pi). Eqn. 7 calculates the distance between the two embedding vectors. Examiner considers the anomaly score as determining whether the product is defective).
Claims 5-7 and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (Kim, Daehwan, et al. "Spatial contrastive learning for anomaly detection and localization." IEEE Access 10 (2022): 17366-17376) (hereafter, “Kim”) (IDS) in view of Chen et al. (US 2021/0105578) (hereafter, “Chen”) as applied to claims 1 and 10 above, and further in view of Schroff et al. (Schroff, Florian, Dmitry Kalenichenko, and James Philbin. "FaceNet: A unified embedding for face recognition and clustering." 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015) (hereafter, “Schroff”).
Regarding claim 5, in which claim 1 is incorporated, Kim discloses wherein the learning processor is configured to train a feature extraction model by inputting the second normal product image belonging to the normal classification, [the abnormal modified image], and the abnormal restored image as learning data into the feature extraction model which outputs an expression vector for the input image data (Page 17369, right column, paragraph 1-2, instead of using images with different applications of augmentation as positive, we use the exact same location as the original image restored by autoencoder as positive … pairs of input
patches Pi(i D 1; : : : ;N) and reconstructed patches Qj(j D 1; : : : ;N)).
However, neither Kim nor Chen, explicitly disclose the abnormal modified image.
Schroff teaches the abnormal modified image (Page 817, left column, paragraph 1, we strive for an embedding f(x), … whereas the squared distance between a pair of face images from different identities is large. Examiner considers images with different identities, in combination with Kim, to teach including the abnormal modified image as it includes defects not present in the normal or restored images).
Kim, Chen, and Schroff are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Schroff is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature extraction of Schroff into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to improved efficiency, as suggested by Schroff at Abstract, the benefit of our approach is much greater representational efficiency: we achieve state-of-the-art face recognition performance using only 128-bytes per face.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Schroff.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Schroff to obtain the invention as specified in claim 5.
Regarding claim 6, Kim in view of Chen discloses the artificial intelligence device of claim 5.
However, neither Kim nor Chen, explicitly disclose wherein the learning processor is configured to perform contrastive learning on the feature extraction model so that a distance between an expression vector of the second normal product image belonging to the normal classification and an expression vector of the abnormal restored image is less than a distance between an expression vector of the abnormal modified image and the expression vector of the abnormal restored image.
Schroff teaches wherein the learning processor is configured to perform contrastive learning on the feature extraction model so that a distance between an expression vector of the second normal product image belonging to the normal classification and an expression vector of the abnormal restored image is less than a distance between an expression vector of the abnormal modified image and the expression vector of the abnormal restored image (Fig. 3; Page 817, left column, paragraph 1, we strive for an embedding f(x), such that the squared distance between all faces, independent of imaging conditions, of the same identity is small, whereas the squared distance between a pair of face images from different identities is large).
Kim, Chen, and Schroff are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Schroff is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature extraction of Schroff into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to improved efficiency, as suggested by Schroff at Abstract, the benefit of our approach is much greater representational efficiency: we achieve state-of-the-art face recognition performance using only 128-bytes per face.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Schroff.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Schroff to obtain the invention as specified in claim 6.
Regarding claim 7, Kim in view of Chen discloses the artificial intelligence device of claim 5.
However, neither Kim nor Chen, explicitly disclose wherein the learning processor is configured to input the second normal product image belonging to the normal classification as positive sample input data into the feature extraction model, input the abnormal modified image as negative sample input data into the feature extraction model, input the abnormal restored image as anchor input data into the feature extraction model, and train the feature extraction model through a triplet loss function.
Schroff teaches wherein the learning processor is configured to input the second normal product image belonging to the normal classification as positive sample input data into the feature extraction model, input the abnormal modified image as negative sample input data into the feature extraction model, input the abnormal restored image as anchor input data into the feature extraction model (Eqn. 2; Page 817, left column, paragraph 3, Here we want to ensure that an image xa i (anchor) of a specific person is closer to all other images xp i (positive) of the same person than it is to any image xn i (negative) of any other person), and train the feature extraction model through a triplet loss function (Eqn. 2; Page 817, left column, paragraph 2, we believe that the triplet loss is more suitable).
Kim, Chen, and Schroff are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Schroff is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature extraction of Schroff into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to try different configurations of image types as there is a finite number of ways to select which of the images (normal, abnormal, and restored abnormal) to be the anchor, positive, and negative images. Further, the different configurations of image types would have yielded predictable results with a reasonable expectation of success.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Schroff.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Schroff to obtain the invention as specified in claim 7.
Regarding claim 14, in which claim 10 is incorporated, Kim discloses training, by the learning processor, a feature extraction model by inputting the second normal product image belonging to the normal classification, [the abnormal modified image], and the abnormal restored image as learning data into the feature extraction model which outputs an expression vector for the input image data (Page 17369, right column, paragraph 1-2, instead of using images with different applications of augmentation as positive, we use the exact same location as the original image restored by autoencoder as positive … pairs of input
patches Pi(i D 1; : : : ;N) and reconstructed patches Qj(j D 1; : : : ;N)).
However, neither Kim nor Chen, explicitly disclose the abnormal modified image.
Schroff teaches the abnormal modified image (Page 817, left column, paragraph 1, we strive for an embedding f(x), … whereas the squared distance between a pair of face images from different identities is large. Examiner considers images with different identities, in combination with Kim, to teach including the abnormal modified image as it includes defects not present in the normal or restored images).
Kim, Chen, and Schroff are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Schroff is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature extraction of Schroff into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to improved efficiency, as suggested by Schroff at Abstract, the benefit of our approach is much greater representational efficiency: we achieve state-of-the-art face recognition performance using only 128-bytes per face.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Schroff.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Schroff to obtain the invention as specified in claim 14.
Regarding claim 15, Kim in view of Chen discloses the artificial intelligence device of claim 14.
However, neither Kim nor Chen, explicitly wherein the step of training the feature extraction model includes the step of performing contrastive learning on the feature extraction model so that a distance between an expression vector of the second normal product image belonging to the normal classification and an expression vector of the abnormal restored image is less than a distance between an expression vector of the abnormal modified image and the expression vector of the abnormal restored image.
Schroff teaches wherein the step of training the feature extraction model includes the step of performing contrastive learning on the feature extraction model so that a distance between an expression vector of the second normal product image belonging to the normal classification and an expression vector of the abnormal restored image is less than a distance between an expression vector of the abnormal modified image and the expression vector of the abnormal restored image (Fig. 3; Page 817, left column, paragraph 1, we strive for an embedding f(x), such that the squared distance between all faces, independent of imaging conditions, of the same identity is small, whereas the squared distance between a pair of face images from different identities is large).
Kim, Chen, and Schroff are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Schroff is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature extraction of Schroff into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to improved efficiency, as suggested by Schroff at Abstract, the benefit of our approach is much greater representational efficiency: we achieve state-of-the-art face recognition performance using only 128-bytes per face.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Schroff.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Schroff to obtain the invention as specified in claim 15.
Regarding claim 16, Kim in view of Chen discloses the artificial intelligence device of claim 14.
However, neither Kim nor Chen, explicitly disclose inputting the second normal product image belonging to the normal classification as positive sample input data into the feature extraction model; inputting the abnormal modified image as negative sample input data into the feature extraction model; inputting the abnormal restored image as anchor input data into the feature extraction model; and training the feature extraction model through a triplet loss function.
Schroff teaches inputting the second normal product image belonging to the normal classification as positive sample input data into the feature extraction model; inputting the abnormal modified image as negative sample input data into the feature extraction model; inputting the abnormal restored image as anchor input data into the feature extraction model; (Eqn. 2; Page 817, left column, paragraph 3, Here we want to ensure that an image xa i (anchor) of a specific person is closer to all other images xp i (positive) of the same person than it is to any image xn i (negative) of any other person), and training the feature extraction model through a triplet loss function (Eqn. 2; Page 817, left column, paragraph 2, we believe that the triplet loss is more suitable).
Kim, Chen, and Schroff are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Schroff is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature extraction of Schroff into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to try different configurations of image types as there is a finite number of ways to select which of the images (normal, abnormal, and restored abnormal) to be the anchor, positive, and negative images. Further, the different configurations of image types would have yielded predictable results with a reasonable expectation of success.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Schroff.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Schroff to obtain the invention as specified in claim 16.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (Kim, Daehwan, et al. "Spatial contrastive learning for anomaly detection and localization." IEEE Access 10 (2022): 17366-17376) (hereafter, “Kim”) (IDS) in view of Chen et al. (US 2021/0105578) (hereafter, “Chen”) as applied to claims 8 and 17 above, and further in view of Larson et al. (US 2020/0193331) (hereafter, “Larson”).
Regarding claim 9, in which claim 8 is incorporated, Kim discloses [wherein the processor is configured to determine that the product to be inspected is normal when] the distance between the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D
1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Eqn. 7 calculates the distance between the two embedding vectors) [is less than or equal to a predetermined defect reference value, and determine that the product to be inspected is defective when] the distance between the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D
1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Eqn. 7 calculates the distance between the two embedding vectors) [exceeds the predetermined defect reference value].
However, neither Kim nor Chen, explicitly disclose wherein the processor is configured to determine that the product to be inspected is normal when the distance between the first expression vector and the second expression vector is less than or equal to a predetermined defect reference value, and determine that the product to be inspected is defective when the distance between the first expression vector and the second expression vector exceeds the predetermined defect reference value.
Larson teaches wherein the processor is configured to determine that the product to be inspected is normal when the distance between the first expression vector and the second expression vector is less than or equal to a predetermined defect reference value (Fig. 4; ¶0096, metric associated with the instance does not satisfy a dynamic similarity threshold or the like. Fig. 4 illustrates points within a threshold as being normal), and determine that the product to be inspected is defective when the distance between the first expression vector and the second expression vector exceeds the predetermined defect reference value (Fig. 4; ¶0096, metric (e.g., distance value) associated with the instance satisfies or exceeds a dynamic anomaly threshold. Fig. 4 illustrates points outside a threshold as being abnormal).
Kim, Chen, and Larson are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Larson is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the threshold of Larson into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to rapidly train a model, as suggested by Larson at ¶0061, The method 200 functions to enable a rapid and intelligent training of one or more machine learning models.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Larson.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Larson to obtain the invention as specified in claim 9.
Regarding claim 18, in which claim 17 is incorporated, Kim discloses [determining that the product to be inspected is normal when the distance between] the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D 1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Eqn. 7 calculates the distance between the two embedding vectors) [is less than or equal to a predetermined defect reference value; and determining that the product to be inspected is defective when] the distance between the first expression vector and the second expression vector (Eqn. 7; Page 17370, right column, paragraph 2, The patch set of the original image denotes pi(i D
1; … ;m), and the patch set of the image restored through the autoencoder denotes qj(j D 1; … ;m). Then, two embedding vector sets are generated as fcon(pi) and fcon(qj). Eqn. 7 calculates the distance between the two embedding vectors) [exceeds the predetermined defect reference value].
However, neither Kim nor Chen, explicitly disclose determining that the product to be inspected is normal when the distance between the first expression vector and the second expression vector is less than or equal to a predetermined defect reference value; and determining that the product to be inspected is defective when the distance between the first expression vector and the second expression vector exceeds the predetermined defect reference value.
Larson teaches determining that the product to be inspected is normal when the distance between the first expression vector and the second expression vector is less than or equal to a predetermined defect reference value (Fig. 4; ¶0096, metric associated with the instance does not satisfy a dynamic similarity threshold or the like. Fig. 4 illustrates points within a threshold as being normal); and determining that the product to be inspected is defective when the distance between the first expression vector and the second expression vector exceeds the predetermined defect reference value (Fig. 4; ¶0096, metric (e.g., distance value) associated with the instance satisfies or exceeds a dynamic anomaly threshold. Fig. 4 illustrates points outside a threshold as being abnormal).
Kim, Chen, and Larson are analogous to the claimed invention because Kim is directed towards anomaly detection methods, Chen is directed towards models with reconstruction networks, and Larson is directed towards image classification. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the threshold of Larson into the multiple processors and data augmentation of Chen and the model of Kim. The suggestion/motivation for doing so would have been to rapidly train a model, as suggested by Larson at ¶0061, The method 200 functions to enable a rapid and intelligent training of one or more machine learning models.
This method of improving Kim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Larson.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Kim with the teachings of Chen and Larson to obtain the invention as specified in claim 18.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Manoochehri et al. (US 2023/0215145) discloses using a threshold for determining classes (¶0082, a contrastive learning model, etc. In some implementations, the feature extractor may be configured to identify features which are close together as similar (e.g., closer than a threshold value) and features which are far apart as dissimilar (e.g., further apart than the threshold value)).
Chopde et al. (US 2022/0188577) discloses data augmentation methods (¶0013, The augmentation module may comprise cropping and resizing, rotation and cutout, color distortions, Gaussian blur, and Sobel filtering).
Ye et al. (Ye, Fei, et al. "Attribute restoration framework for anomaly detection." IEEE Transactions on Multimedia 24 (2020): 116-127) discloses restoration based anomaly detection network (Fig. 2; Page 120, right column, last paragraph, Both normal and anomalous data are fed into the model, which are utilized together to deter mine whether a query sample is anomalous).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMAO DING whose telephone number is (571)272-7237. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAOMAO DING/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights