Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The Restriction on Group I has been withdrawn.
Application has new claim 20, thus the application has pending claims 1-20.
Claim Objections
Claims 1, 11, and 13 are objected to because of the following informalities: “inputting the test query set into a deep learning neural network…”. It is unclear if “a deep neural network” is in reference to a deep neural network in the “selecting a test query set step”. For the purposes of examination, Examiner is understanding the neural network of the selecting step is used just to describe the purpose of the test query set of images. Appropriate correction is required.
Claim 2 recites the limitation “…with symbolic anomalies,” but the symbolic anomalies were applied in the beginning of the claim. Examiner suggests clarifying the language to say “the symbolic anomalies” to establish antecedent basis.
Claim 5 recites the limitation “the ROI data”. There is insufficient antecedent basis for this limitation in the claim, as the claim stated “a ROI dataset” prior. For purposes of examination, the Examiner is interpreting the ROI data to be “the ROI dataset”. Appropriate correction is required.
Claim 9 objected to because of the following informalities: “iterated ROI and the iterated anomaly,”. The comma should be a semi-colon. Appropriate correction is required.
Claims 11 recites the limitation “rating a similarity distance score between the at least one image…”. There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, Examiner is interpreting this to be the at least one manufacturing product image. Claim 13 is also objected for the similar limitation “rate a similarity distance score between the at least one image…”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 8 recites the limitations “the images” and “the dataset”. There is insufficient antecedent basis for this limitation in the claim. It is unclear which images “the images” are referring to. Additionally, it is unclear which dataset is being referred to.
Allowable Subject Matter
Claims 3-4 and 6-10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Barath BALASUBRAMANIAN US-20220171995-A1, hereinafter Barath, in further view of Yujiao Cheng US-20240412349-A1, hereinafter Cheng.
As per claim 1, Barath discloses a method of training a neural network to detect anomalies in a manufacturing product, comprising (see Barath ¶33, wherein a method to detect anomalies is disclosed. See also ¶41, wherein training a machine learning model, such as a neural network, is disclosed): obtaining a dataset including multiple manufacturing product images and annotation files with coordinates of predetermined anomalous regions corresponding to the multiple manufacturing product images (see Barath ¶60, wherein a dataset including labeled and unlabeled data of objects is disclosed. The images in the dataset are of machine parts, i.e., manufactured product, as described in ¶34-36. The labeled dataset includes annotations of the anomaly. It is further described in ¶100-102 and FIG. 16, the locations of these anomalies in the annotation), respectively;selecting a test query set of images and a support set of images from the dataset multiple times to train a deep learning neural network (see Barath ¶67 and FIG. 4, wherein training of the machine learning model is performed on the testing dataset, i.e., test query set, which includes the annotated set from anomaly labeling, i.e., support set. This is done for multiple iterations of re-training as also described in ¶143. The machine learning model can also be a deep learning model, such as a deep learning neural network, as disclosed in ¶225), the support set of images including pairs of reference images and each pair including at least one anomalous manufacturing product image and at least one non-anomalous manufacturing product image (see Barath ¶67, wherein the annotated dataset includes anomalous and normal images).
While Barath does disclose a ranking score for a trained model (see Barath ¶147-149, wherein a trained model generates a score using an annotated dataset), Barath fails to explicitly disclose where Cheng teaches:inputting the test query set into a deep learning neural network to compare the pairs of reference images (see Cheng ¶98 and FIG. 12, wherein a plurality of images, i.e., the test query set, is input into a neural network. The input 1210, which contains good and bad images, is compared with the memory bank 340, which distinguishes the good parts or anomaly parts, i.e., reference images, as discussed in ¶38);rating, with the deep learning neural network, a similarity score based on a similarity distance between the pairs of reference images of the support set of images based on the labeled files (see Cheng ¶98-99, wherein a weighted distance is calculated using the feature extractor and the memory bank along with the label for each image, i.e., labeled set, to acquire the anomaly score, i.e., similarity score based on similarity distance. The feature extractor included in this process is CNN-based, i.e., deep learning neural network);determining whether each image of the pairs of reference images is anomalous based on the similarity distance score (see Cheng ¶101-102 and FIG. 12, wherein the image is classified as a good image (-1) or an anomalous image (1));and adjusting parameters characterizing the deep learning neural network through a model based on the similarity score (see Chang ¶108 and FIG. 13, wherein a weight vector w is updated, as in adjusted, based on the lore score bad parts and high score good parts of step 1312. This is performed during training mode as disclosed in ¶104. The feature data is acquired from the feature extractor, which is disclosed to be CNN-based in Chang ¶98).
While Cheng discloses labeled files, both Barath and Cheng use a labeled or annotated set in order to acquire a ranked anomaly score. Thus, it would have been obvious to use the labeled files as the annotated set disclosed in Barath, which contains the coordinates in order to determine the precise location of the defects. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Barath’s method by using Cheng’s teaching by inputting the test query set into the deep learning neural network in order to improve the neural network’s detection of defects by utilizing a set of normal and defect images.
As per claim 19, Barath in combination with Cheng, discloses a non-transitory computer readable medium having computer readable instructions stored thereon which, when executed on a processor, cause a computer to perform a method as defined in claim 1 (see Barath ¶33, wherein a non-transitory computer readable storage media for performing defect detection is disclosed).
Claims 2 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Barath, in combination with Cheng, in further view of Saksham Jain Synthetic data augmentation, hereinafter Jian, and Ruikang Liu Anomaly-GAN, hereinafter Liu.
As per claim 2, Barath, in combination with Cheng, fails to explicitly disclose where Jian teaches:The method according to claim 1, further comprising:generating synthetic data by processing a raw dataset to apply symbolic anomalies on the multiple manufacturing product images (see Jian page 6/14, wherein synthetic data is generated, which are fakes that resemble the input images, using a GAN. The GAN, namely a DCGAN, is capable of generating defects, i.e., symbolic anomalies, on the input images of the manufactured article (steel strips, as described in page 3) at different angles and severities).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Barath’s, in combination with Cheng, method by using Jian’s teaching by generating synthetic data to the multiple manufacturing product images in order to create a larger set of training images.
While Barath, in combination with Cheng and Jian, applies synthetic data into a deep learning neural network (see Jian page 4/14, where the CNN requires much data which is why they use the synthetic data augmentation approach as further described in page 6/14, i.e., the synthetic data is sent into the CNN), it fails to explicitly disclose where Liu teaches: applying the synthetic data into the deep learning neural network to identify relevant regions of interest (ROls) (see Liu pages 5-7 Section 4.2.1, wherein generated anomaly images are fed into an anomaly aware network which calculates, as in identifies, the bounding box around the anomaly region. see further Liu page 9/16 wherein it is clarified that the generated anomaly images are feed into a R-CNN), of each one of the multiple manufacturing product images with symbolic anomalies (see Liu FIG. 8, wherein the different manufacturing products contains generated anomalies).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Barath’s, in combination with Cheng and Jian, by using Liu’s teaching by applying the synthetic data to the neural network in order to further train the neural network with more data so that the network can more quickly find the regions contain defects within an image.
As per claim 5, Barath, in combination with Cheng, Jian, and Liu, discloses the method according to claim 2, further comprising: processing the synthetic data to generate a ROI dataset based on the annotation files and the raw dataset (see Liu page 4/16 and FIG. 1, wherein a surface anomaly dataset is generated using the normal and anomaly datasets, i.e., raw and annotated sets. The bounding boxes, i.e., ROI, are calculated around the anomaly), wherein the ROI data includes two sets of ROls with anomalous and non-anomalous ROls, respectively (see Liu FIG. 2, wherein the bounding box around the missing screws and normal screws are found).
Claims 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over Barath, in further view of Cheng and Yehonatan Hai OFIR US-20240338811-A1, hereinafter OFIR.
As per claim 11, Barath discloses a method of detecting anomalies in a manufacturing product by using a deep learning neural network trained with a dataset comprising images of different manufacturing products, comprising:obtaining at least one image of a manufacturing product to be inspected (see Barath ¶60, wherein a testing dataset including labeled and unlabeled data of objects is disclosed. The images in the dataset are of machine parts, i.e., manufactured product, as described in ¶34-36);inputting the at least one image obtained in a neural network trained with a dataset including multiple manufacturing product images and annotation files with coordinates of predetermined anomalous regions corresponding to the multiple manufacturing product images, respectively (see Barath ¶67 and FIG. 4, wherein training is performed on the testing dataset, i.e., test query set, which includes the annotated set, which contains anomalous and normal images, from anomaly labeling, i.e., support set. It is further described in ¶100-102 and FIG. 16, the locations of these anomalies in the annotation. The training a machine learning model, such as a neural network, is disclosed in Barath ¶41);wherein the training of the neural network comprises:selecting a test query set of images and a support set of images from the dataset multiple times to train a deep learning neural network (see Barath ¶67 and FIG. 4, wherein training is performed on the testing dataset, i.e., test query set, which includes the annotated set from anomaly labeling, i.e., support set. This is done for multiple iterations of re-training the machine learning model, described in ¶41, as described in ¶143. The machine learning model can also be a deep learning model, such as a deep learning neural network, as disclosed in ¶225), the support set of images including pairs of reference images and each pair including at least one anomalous manufacturing product image and at least one non-anomalous manufacturing product image (see Barath ¶67, wherein the annotated dataset includes anomalous and normal images).
While Barath does disclose a ranking score for a trained model (see Barath ¶147-149, wherein a trained model generates a score using an annotated dataset), Barath fails to explicitly disclose where Cheng teaches:inputting the test query set into a deep learning neural network to compare the pairs of reference images (see Cheng ¶98 and FIG. 12, wherein a plurality of images, i.e., the test query set, is input into a feature extractor, which is a convolutional neural network. The input 1210, which contains good and bad images, is compared with the memory bank 340, which distinguishes the good parts or anomaly parts, i.e., reference images, as discussed in ¶38);rating, with the deep learning neural network, a similarity score based on a similarity distance between the pairs of reference images of the support set of images based on the labeled files (see Cheng ¶98-99, wherein a weighted distance is calculated using the feature extractor and the memory bank along with the label for each image, i.e., annotated set, to acquire the anomaly score, i.e., similarity score based on similarity distance. The feature extractor included in this process is CNN-based, i.e., deep learning neural network);determining whether each image of the pairs of reference images is anomalous based on the similarity distance score (see Cheng ¶101-102 and FIG. 12, wherein the image is classified as a good image (-1) or an anomalous image (1)); andadjusting parameters characterizing the deep learning neural network through a model based on the similarity score (see Chang ¶108 and FIG. 13, wherein a weight vector w is updated, as in adjusted, based on the lore score bad parts and high score good parts of step 1312. This is performed during training mode as disclosed in ¶104. The feature data is acquired from the feature extractor, which is disclosed to be CNN-based in Chang ¶98);comparing a manufacturing product image with reference images, the reference images being stored as a support set of images (see Cheng ¶99, wherein the weighted k-NN module compares the image features of the input set, which contains labeled anomalous and non-anomalous images, to the image features in the memory bank, i.e., support set, which contains non-anomalous image);rating a similarity distance score between the at least one image obtained and the reference images (see Cheng ¶101-102 and FIG. 12, wherein the image is rated 1 or -1 as a good image (-1) or an anomalous image (1). In FIG. 12, it is shown that this is done in the online test, i.e., outside the training); anddetermining whether the manufacturing product is anomalous based on the similarity distance score (see Cheng ¶101-102 and FIG. 12, wherein the image is determined as a good or an anomalous).
While Cheng discloses labeled files, both Barath and Cheng use a labeled or annotated set in order to acquire a ranked anomaly score. Thus, it would have been obvious to use the labeled files as the annotated set disclosed in Barath, which contains the coordinates in order to determine the precise location of the defects. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Barath’s method by using Cheng’s teaching by inputting the test query set into the deep learning neural network in order to improve the neural network’s detection of defects by utilizing a set of normal and defect images.
However, while Barath, in combination with Cheng, discloses the reference images in the support set comprising anomalous and non-anomalous (see Cheng ¶99, wherein the anomalous and non-anomalous image set is disclosed), it fails to explicitly disclose that the reference images are pairs. OFIR teaches:pairs of reference images including at least one anomalous manufacturing product image and at least one non-anomalous manufacturing product image (see OFIR ¶14 and more specifically ¶99-100, wherein a ML autoencoder compares an input image with nominal images. The nominal images are training images, which, as further disclosed in ¶102-104, the training images can be a pair comprising a ground-truth reference image and a defective image, i.e., anomalous and non-anomalous images).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Barath’s, in combination with Cheng, method by using OFIR’s teaching by specifying that the reference images are a pair of anomalous and non-anomalous images to the reference images in order to help the neural network better compare defective and normal products.
As per claim 12, Barath, in combination with Cheng and OFIR, discloses the method according to claim 11, further comprising: setting another support set comprising images of a different version of the manufacturing product (see Cheng ¶98, wherein a labeled bank of image parts with known classification is disclosed. The parts are good and bad versions of the same subject as disclosed in ¶110 and FIG. 2).
As per claims 13-14, the rationale provided in claims 11-12 is incorporated herein. The method of claims 11-12 corresponds to the system of claims 13-14.
As per claim 15, Barath in combination with Cheng and OFIR, discloses the inspection system according to claim 14, further comprising an additional deep learning neural network configured as an object detector module (ODM) to identify relevant regions of interest (ROls) in multiple manufacturing product images (see OFIR ¶101-103, wherein a second ML model is used to generate defect images which contain regions of defective features corresponding to the reference image. The first ML model, which relates to the additional deep learning neural network, detects these defects generated by the second ML, as disclosed in ¶110-111).
As per claim 16, Barath in combination with Cheng and OFIR, discloses the inspection system according to claim 13, wherein the memory device stores a deep learning neural network and the support set of images (see Cheng ¶96 and FIG. 12, wherein training data stored in a memory bank and coupled with CNN-based feature extractor).
As per claim 17, Barath in combination with Cheng and OFIR, discloses the inspection system according to claim 16, wherein the deep learning neural network returns an output tensor for each query image (see Cheng ¶87-88 and FIG. 10, wherein the output of the CNN for the input images is disclosed), and the output tensor comprises coordinates of all detected objects and a classification inference result based on the similarity score between each query image and the support set of images (see Cheng ¶101, wherein a classification is based on an anomaly score using a threshold to determine if the part is good or bad. The threshold is determined by the optimal weighting values which were calculated with the labeled images, i.e., support set. See also OFIR ¶54, wherein the output also includes the locations of the defects).
As per claim 18, Barath in combination with Cheng and OFIR, discloses the inspection system according to claim 13, comprising an acquisition booth (I) to automate the inspection and including an imaging system, wherein the imaging system includes a photographic camera, an illumination system and sensors (see Cheng ¶119, wherein an imaging system, which includes a camera, is used to provide images for a fully automated system. Other sensors can be included in this system as disclosed in Cheng ¶27. Further configurations of the image, such as lighting, can be further done as disclosed in Barath ¶116).
As per claim 20, Barath in combination with Cheng and OFIR, discloses a non-transitory computer readable medium having computer readable instructions stored thereon which, when executed on a processor, cause a computer to perform a method as defined in claim 11 (see Barath ¶33, wherein a non-transitory computer readable storage media for performing defect detection is disclosed).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bradley Obas Felix whose telephone number is (703)756-1314. The examiner can normally be reached M-F 8-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 5712728243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRADLEY O FELIX/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights