DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicant
Limitations appearing inside of {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Claims 1-21 are pending in the application.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 17 and 21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 17 and 21 recite “scratch and/or defect”. It is unclear what is required for the claimed invention. For the purpose of examination the limitation will be interpreted as “scratch or defect”
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 4, 9-12, and 19-21 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by “Dong” (Z. Dong, X. Shao and R. Zhang, "Surface Defect Segmentation with Multi-column Patch-Wise U-net," 2019 IEEE 5th International Conference on Computer and Communications (ICCC), Chengdu, China, 2019, pp. 1436-1441, doi: 10.1109/ICCC47050.2019.9064246.).
Regarding claim 1, Dong teaches a learning method executed by a learning apparatus including a processor (Nvidia GeForce GTX 1070 GPU; Dong, [p 1439, Col 1, §IV, ¶1]), the learning method comprising: causing the processor to execute:
a data acquisition step of acquiring learning data consisting of a pair of a patch image and correct answer data of a class label for a unit region of the patch image (Our method takes the full surface defect images as input, divides them into some patches randomly; Dong, [p1473, Col 2, §III. A., ¶1]; Our model is trained with only ellipse labels to predict defect/non-defect areas and evaluated its performance on the test dataset with pixel level annotations; [p1473, Col 2, §III, ¶1].);
a determination step of performing segmentation of the patch image by using a learning model and the learning data (we propose a new Multi-Column CNN with three patch-based U-net modules (see Figure 3) which uses filters with different sizes (3x3,5x5,7x7) of local receptive field to learn the defect segmentation mask, larger receptive fields can be useful for segmenting larger defect regions; Dong [p 1438, Col 1, §III. B., ¶2]), and
determining, for each patch image, whether or not a second unit region is correctly detected by the learning model (once the mean of the pixel level (corresponding to the instance level) probabilities p_ij surpasses a certain threshold, a patch level (corresponding to the bag level) probability p_ij is activated; Dong, [p 1438, Col 2, §III. B., ¶2]);
a weighting step of setting a first weight based on a result of the determination (we add a class-balancing weight [Symbol font/0x62] to balance the defect and non-defect classes; Dong, [p 1439, Col 1, §III. B., ¶3]); and
an update step of updating the learning model based on a result of the weighting (Eq 3 exhibits the loss function dependent on the weighting [Symbol font/0x62], which is used to update the model according to the weight during backpropagation; Dong, [p 1439, Col 1, §III.B., ¶3]
PNG
media_image1.png
294
470
media_image1.png
Greyscale
).
Regarding claim 2, Dong teaches the learning method according to claim 1. Dong further teaches wherein, in the weighting step, the processor sets the first weight, in a unit of the patch image, for a first loss that is a loss for each patch image (we add a class-balancing weight [Symbol font/0x62] to f(x), f(x) is the model’s predicted probability for the defect pixel ;Dong, [p 1439, Col 1, §III.B., ¶3]).
Regarding claim 4, Dong teaches the learning method according to claim 1. Dong further teaches wherein, in the determination step, the learning model detects the second unit region belonging to a specific class (So the model with MIL layer accurately classifies regions with 9,2,4 and 3 (i.e., specific classes); Dong, [p 1440-41, Col 2, §IV. Ablation Experiments, ¶2]).
Regarding claim 9, Dong teaches the learning method according to claim 4. Dong further teaches wherein the learning model outputs a certainty of the detection (We add MIL layer behind the last convolutional layer of our network, it computes probability map and each element represents the probability pij of the category in which it falls according Eq. (1).; Dong, [p 1438, Col 1, §III.B., ¶1]), and
in the determination step, the processor determines whether or not the second unit region belongs to the specific class based on whether or not the certainty is equal to or higher than a threshold value (once the mean of the pixel level (corresponding to the instance level) probabilities pij surpasses a certain threshold, a patch level (corresponding to the bag level) probability pij is activated; Dong, [p 1438, Col 2, §III.B. MIL Layer, ¶2]).
Regarding claim 10, Dong teaches the learning method according to claim 9. Dong further teaches wherein the processor changes the threshold value in a process of learning (MIL layer is used to activate a patch level class probability pij by softmax function when the mean of the instance level probabilities surpasses the setting threshold. Because we set the feature tensors [ch,w, h] to [ch,wxh], the value of fij is computed in the global scope of image patch. The parameters a and bij control the shape of sigmoid function, [Symbol font/0x20][Symbol font/0x73](abij) and [Symbol font/0x20][Symbol font/0x73](-abij) ab are added to normalize patch probability pij to [0,1]; Dong, [p 1438, Col 2, §III.B. MIL Layer, ¶3]).
Regarding claim 11, Dong teaches the learning method according to claim 1,
wherein, in the weighting step, the processor performs the weighting on a cross-entropy loss of the patch image (See Dong, eq. 3, shown below, exhibits weighted cross entropy loss function; [p 1439, Col 1, §III.B. Weighted SoftMax Loss, ¶2]
PNG
media_image2.png
292
491
media_image2.png
Greyscale
).
Regarding claim 12, Dong teaches the learning method according to claim 1. Dong further teaches wherein the processor further executes a loss function derivation step of deriving a loss function for a batch composed of the patch images (See Dong, Figure 2, shown below, exhibits the output of the class scores matrix is batch sized, and is input into the Weight SoftMax Loss function
PNG
media_image3.png
386
695
media_image3.png
Greyscale
), and
updates the learning model by using the loss function in the update step (loss function is understood by one of ordinary skill in the art to update the model during backpropagation).
Regarding claim 15, Dong teaches the learning method according to claim 1. Dong further teaches wherein, in the update step, the processor updates a parameter of the learning model to minimize the loss function (one of ordinary skill in the art would understand that a learning network, as taught by Dong, would update parameters during backpropagation to minimize the loss).
Regarding claim 16, Dong teaches the learning method according to claim 1. Dong further teaches wherein, in the data acquisition step, the processor inputs an image to acquire a divided image of the input image as the patch image (Our method takes the full surface defect images as input, divides them into some patches randomly and inputs them to a multi-column FCN; Dong, [p 1437, Col 2, §III, ¶1]).
Regarding claim 17, Dong teaches the learning method according to claim 1. Dong further teaches wherein, in the data acquisition step, the processor acquires the patch image of a size corresponding to a size of a scratch and/or a defect of a subject to be detected (larger receptive fields can be useful for segmenting larger defect regions; Dong, [p 1438, Col 2, §III.B. Multi-Column CNN Model, ¶1]
PNG
media_image4.png
232
469
media_image4.png
Greyscale
).
Regarding claim 18, Dong teaches the learning method according to claim 1. Dong further teaches wherein the learning model includes a neural network that performs the segmentation (we rely on Fully Convolutional Neural Network (FCN) [3], an important segmentation method; Dong, [p 1436, Col 2, §I, ¶2]).
Claim 19 is similarly analyzed as analogous claim 1.
Regarding claim 20, Dong teaches a non-transitory, computer-readable tangible recording medium on which a program for causing, when read by a computer, a processor provided to the computer to execute the learning method according to claim 1 is recorded (Our patch-based model is implemented in python with Keras and TensorFlow libraries, we ran the experiment within 1 day on Nvidia GeForce GTX 1070 GPU, which requires storage; Dong, [p 1439, Col 1, §IV, ¶1]).
Claim 21 is similarly analyzed as claim 1. Dong further teaches wherein the trained model is used to detect a scratch and/or a defect of a subject from an input image (Our method takes the full surface defect images as input,… Our model is trained… to predict defect/non-defect areas; Dong, [p 1437, Col 2, §III, ¶1]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Dong, in view of, Suzuki, US20200117991A1, as cited in the IDS.
Regarding claim 3, Dong teaches the learning method according to claim 1. Dong further teaches wherein, in the weighting step, the processor sets, as the first weight, a larger weight in a case in which it is determined that the second unit region is not correctly detected than in a case in which it is determined that the second unit region is {correctly detected} (Dong teaches 2 cases, defect and non-defect, and they weight the loss contribution of false negative and false positive (i.e., not correctly detected) examples by our Weighted SoftMax loss function; Dong, [Abstract]). However Dong does not explicitly teach as the first weight, a larger weight in a case in which it is determined that the second unit region is not correctly detected than in a case in which it is determined that the second unit region is correctly detected
However, Suzuki, a similar field of endeavor of detection and misclassification of objects, teaches as the first weight, a larger weight in a case in which it is determined that the second unit region is not correctly detected than in a case in which it is determined that the second unit region is correctly detected (The machine learning apparatus 100 compares the obtained results against correct regions and classes indicated by training information attached to the training dataset, to thereby calculate error over all the detected region proposals. The machine learning apparatus 100 updates the synaptic weights in the detection model to reduce the error; Suzuki, ¶[0118]).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Dong in view of Suzuki, and further in view of Ikeda et al., US 20190147586 A1.
Regarding claim 5, Dong teaches the learning method according to claim 4. Dong does not explicitly disclose wherein, in the determination step, the processor determines that the second unit region is not correctly detected in a first case in which the second unit region belonging to the specific class is erroneously detected by the learning model and in a second case in which the second unit region belonging to the specific class is not detectable by the learning model.
However, Suzuki wherein, in the determination step, the processor determines that the second unit region is not correctly detected (the machine learning apparatus 100 compares the obtained results against correct regions and classes indicated by training information attached to the training dataset, to thereby calculate error over all the detected region proposals; Suzuki, ¶[0118]) in a first case in which the second unit region belonging to the specific class is erroneously detected by the learning model (The first term of the error modification value L.sub.mod represents a difference between the feature confidence measure vector 74 and a correct class vector 75, which indicates that the classes C1 and C2 are incorrect and the class C3 is correct; Suzuki, ¶[0133]).{and in a second case in which the second unit region belonging to the specific class is not detectable by the learning model}.
Suzuki does not explicitly disclose and in a second case in which the second unit region belonging to the specific class is not detectable by the learning model.
However, Ikeda, a similar field of endeavor, teaches in a second case in which the second unit region belonging to the specific class is not detectable by the learning model (in the case where an undetected defect has occurred a predetermined number of times, the filter parameter of the aforesaid preprocessing filter may be updated based on an image (hereinafter also “non-detection image”) including the undetected defect; Ikeda, ¶[0117]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include an undetected defects case as taught by Suzuki to the invention of Dong. The motivation to do so would be to improve the class determination accuracy of the detection model.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include an undetected defect case as taught by Ikeda to the combined invention of Dong and Suzuki. The motivation to do so would be to update the processing filter with false negative cases.
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Dong in view of Suzuki, in view of Ikeda and further in view of “He” (H. He and E. A. Garcia, "Learning from Imbalanced Data," in IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263-1284, Sept. 2009, doi: 10.1109/TKDE.2008.239.).
Regarding claim 6, the combination of Dong, Suzuki, and Ikeda teach the learning method according to claim 5. The combination does not explicitly disclose wherein, in the weighting step, the processor sets a larger weight in the second case than in the first case.
However, He, a similar field of endeavor of learning from imbalanced data, teaches wherein, in the weighting step, the processor sets a larger weight in the second case than in the first case (He [p 1270, col 2, §3.2.1, ¶1]; in a binary classification scenario, we define CðMin;MajÞ as the cost of misclassifying a majority class example as a minority class example and let CðMaj;MinÞ represents the cost of the contrary case).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include a larger weight in the second case than the first as taught by He to the combined invention Dong, Suzuki, and Ikeda. The motivation to do so would be to minimize the overall cost on a data set based on probabilities/risk and select the best training data for induction.
Regarding claim 7, the combination of Dong, Suzuki, and Ikeda teaches The learning method according to claim 5. The combination does not explicitly disclose wherein, in the determination step, the processor determines that a result of the detection is correct in a third case in which the result of the detection is neither the first case nor the second case.
However He teaches wherein, in the determination step, the processor determines that a result of the detection is correct in a third case in which the result of the detection is neither the first case nor the second case (He [p 1270, col 2, §3.2.1, ¶1]; Typically, there is no cost for correct classification of either class and the cost of misclassifying minority examples is higher than the contrary case, i.e., CðMaj;MinÞ > CðMin;MajÞ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include correct detection as taught by He to the combined invention of Dong, Suzuki, and Ikeda. The motivation to do so would be to not penalize correct cases.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Dong in view of “Bhatt” (Bhatt, P. M., Malhan, R. K., Rajendran, P., Shah, B. C., Thakar, S., Yoon, Y. J., and Gupta, S. K. (February 9, 2021). "Image-Based Surface Defect Detection Using Deep Learning: A Review." ASME. J. Comput. Inf. Sci. Eng. August 2021; 21(4): 040801. https://doi.org/10.1115/1.4049535).
Regarding claim 8, Dong teaches the learning method according to claim 4. Dong further teaches wherein, in the determination step, the processor performs the determination on {a scratch and} a defect of a subject (surface defect in industrial field (i.e., defect), Dong, [p 1441, Col 1, §V, ¶2). Dong does not explicitly teach determination on scratch and a defect.
However, Bhatt, in a similar field of endeavor of model-based techniques for defect detection in images, teaches wherein, in the determination step, the processor performs the determination on a scratch and a defect of a subject (Surface defects can be of a variety of forms such as scratch, crack, inclusion, spots, dents, holes, and many more…. Concurrent identification of multiple defects problems has been studied in Ref. [34]. The entire system architecture is divided into four stages, (1) anomaly detection, (2) filtering false anomaly, (3) clustering defect pixels, and (4) defect classification; [p 4, §3.3, Col 1, ¶1 and Col 2, ¶2])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include detecting a scratch and a defect as taught by Bhatt to the invention of Dong. The motivation to do so would be because defects arise with various shapes and sizes and there is no need for a custom code needed for training different types of defects. The labeled data for different defects with the appropriate network provides a significantly flexible defect detection mechanism.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Dong in view of Chen et al., CN110728654A.
Regarding claim 13, Dong teaches the learning method according to claim 12. Dong further discloses wherein, in the loss function derivation step, the processor derives, as the loss function, {a first loss function obtained by averaging the result of the weighting over an entire batch composed of the patch images} (a class-balanced soft-max cross-entropy loss, See Eq. 3, [p 1439, Col 1, §III.B. Weighted SoftMax Loss, ¶1]). Dong does not explicitly disclose a first loss function obtained by averaging the result of the weighting over an entire batch composed of the patch images.
However, Chen, a similar field of endeavor of deep convolutional neural network to implement automatic detection and classification of defect types, teaches wherein, in the loss function derivation step, the processor derives, as the loss function, a first loss function obtained by averaging the result of the weighting over an entire batch composed of the patch images (the loss value of each training step is the average of batch_size losses, Chen, ¶[127]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include batch-averaged loss the as taught by Chen to the invention of Dong. The motivation to do so would be to improve computational efficiency and to stabilize the model’s convergence.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Dong in view of Chen and further in view of “Tao” (X. Tao, D. Zhang, W. Hou, W. Ma and D. Xu, "Industrial Weak Scratches Inspection Based on Multifeature Fusion Network," in IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-14, 2021, Art no. 5000514, doi: 10.1109/TIM.2020.3025642: available on ieee xplore on 09/21/2020).
Regarding claim 14, the combination of Dong and Chen teaches the learning method according to claim 13. Chan teaches wherein, in the loss function derivation step, the processor uses, as the loss function, {a function in which the first loss function and }a second loss function, which is a loss function for the batch {and is different from the first loss function, are combined} (the loss value of each training step is the average of batch_size losses, Chen, ¶[127]). The combination does not explicitly disclose a function in which the first loss function and a second loss function, … is different from the first loss function, are combined.
However, Tao, a similar field of endeavor of defect detection imbalanced classes using deep learning, teaches wherein, the loss function, a function in which the first loss function and a second loss function, … is different from the first loss function, are combined (Tao eq. 6 exhibits combined loss function
PNG
media_image5.png
211
441
media_image5.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include combined loss function as taught by Tao to the combined invention of Dong and Chen. The motivation for a combination loss makes the training process converge more stably and robust against the pixels’ imbalanced class population. Therefore, it results in a more precise boundary estimation and predictions on ambiguous scratch areas.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shah et al., US 20200160497 A1, teaches machine based defect detection of three-dimensional (3D) printed objects, and would have been relied up on for teaching updating the model with undetected defect cases.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANDHANA PEDAPATI whose telephone number is 571-272-5325. The examiner can normally be reached M-F 8:30am-6pm (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHANDHANA PEDAPATI/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669