DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Restriction was previously required to one of the following inventions under 35 U.S.C. 121:
I. Claims 2-5 and 12-15, drawn to defect detection that determines feature maps at multiple scales using a multi-scale detection network, classified in G06V10/52.
II. Claims 6-7 and 16-17, drawn to defect detection that determines a standard image using a codec, classified in G06T7/001 and G06N3/0455.
III. Claims 9, 10, and 19, drawn to defect detection that extracts a graph structure feature using a graph convolutional network, classified in G06N7/01.
Claims 1, 8, 11, 18, and 20 are generic with respect to inventions I-III.
In the Reply filed 29 June 2026 Applicant elected Group I without traverse. Therefore, examination will proceed in claims 1-5, 8, 11, 12-15, 18, and 20 with claims 6, 7, 9, 10, 16-17, and 19 being withdrawn as being directed to non-elected invention.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: Circuit Board Defect Determination Using Weighted Sum Of Confidences From Feature Maps At Multiple-Scales and Standard Reference Image Comparison
Drawings
The drawings are objected to because Figs. 4 and 5 are poor quality reproductions that do not have satisfactory reproduction characteristics contrary to 37 CFR 1.84(l). In particular, both of these figures are too small and have poor line quality while Fig. 4 uses fonts that are too small to read. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 11-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Xin (US 2022/0036525 A1) and Hu, Bing, and Jianhui Wang. "Detection of PCB surface defects with improved faster-RCNN and feature pyramid network." IEEE Access 8 (2020): 108335-108345.
Claim 1
In regards to claim 1, Xin discloses a method {Fig. 2A-B and cites below} comprising:
acquiring a
determining a first defect region according to the
determining a second defect region according to the
determining a defect region detection in which the change feature (defect) from the differential image increases the speed of convergence to detect the final defect features as per [0054]. Further, the multi-scale defect detection network uses a synthesized image that includes the differential image; as such the final defect region determination is “according to the first defect region and the second defect region” as broadly recited}.
Hu is an analogous reference in the same field of image-based defect detection. See title, abstract and cites below.
Hu also teaches
acquiring a circuit board image of a circuit board {Fig. 8b image capture of printed circuit board (PCB) image, section IIIF discussing programmable logic controller driving an industrial camera to capture images of the PCB for defect inspection. See also Figs. 4, 5 which input PCB images};
determining a first defect region according to the circuit board image, wherein the first defect region indicates a location of a defect in the circuit board {Figs. 4, 5, 8B, section III-IV in which a feature pyramid network (FPN) that generates feature pyramids of semantic information of the PCB at various scales to determine defect coordinates (location) and type of defect}.
Hu also provides evidence that the claimed “determining a second defect region” is conventionally applied to detecting defects in PCBs using the same process discussed in Xin. See the Section I Introduction and IIA PCB Defects Detection discussing differencing between target image and template image to detect circuit defects such as open circuit, short circuit, etc..
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Xin which already teaches determining a first defect region according to a product image, wherein the first defect region indicates a location of a defect in the product; determining a second defect region according to the product image and a standard image for the product, wherein the second defect region indicates a location of a defect in the product, and the standard image indicates a product without any defect; and determining a defect region of the product according to the first defect region and the second defect region such that the defect detection is for a circuit board as taught by Hu because Hu teaches that Xin’s image differencing defect detection (corresponding to the determining a second defect region) is conventionally applied to circuit boards, because the product being inspected by Xin also requires careful image registration to accurately determine defects just as circuit board defect detection also requires such careful image registration, because Xin motivates using two different techniques in combination to determine defects, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 2
In regards to claim 2, Xin discloses wherein determining the first defect region according to the circuit board image comprises:
determining feature maps at multiple scales of the
determining fusion features of the feature maps at the multiple scales by using a fusion network of the multi-scale detection network {Fig. 2B multi-scale defect detection network, [0053]-[0055] which may be multi-layer convolutional neural network (CNN) that combines (fuses) multi-scale features}; and
determining, according to the fusion features, the first defect region by using a prediction network of the multi-scale detection network {Fig. 2B multi-scale defect detection network, [0053]-[0055] in which the CNN determines a defect map from the combined/fused layers of multi-scale features. It is noted that CNN predicts the output such that the CNN itself is a “prediction network”}.
Hu also teaches
determining feature maps at multiple scales of the circuit board image by using a feature extraction network of a multi-scale detection network {see Figs. 4-6 illustrating residual units (residual blocks) convolutional layers (e.g. Conv1 though Conv5) determining feature maps at multiple scales of the circuit board image};
determining fusion features of the feature maps at the multiple scales by using a fusion network of the multi-scale detection network {see Figs. 4-6. See also IIB discussing fusing information in a multi-scale context}; and
determining, according to the fusion features, the first defect region by using a prediction network of the multi-scale detection network {see Figs. 4-6 illustrating residual units (residual blocks) convolutional layers (e.g. Conv1 though Conv5) determining feature maps at multiple scales of the circuit board image and including concatenated (Concat) residual blocks as further discussed in section III. It is noted that CNN predicts the output such that the CNN itself is a “prediction network”}
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Xin which already teaches determining a first defect region according to a product image including determining feature maps at multiple scales of the product image by using a feature extraction network of a multi-scale detection network; determining fusion features of the feature maps at the multiple scales by using a fusion network of the multi-scale detection network; and determining, according to the fusion features, the first defect region by using a prediction network of the multi-scale detection network such that the such that the defect detection is for a circuit board as taught by Hu because Hu teaches that Xin’s image differencing defect detection (corresponding to the determining a second defect region) is conventionally applied to circuit boards, because the product being inspected by Xin also requires careful image registration to accurately determine defects just as circuit board defect detection also requires such careful image registration, because Xin motivates using two different techniques in combination to determine defects, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 3
In regards to claim 3, Xin is not relied upon to disclose but Hu teaches wherein the feature extraction network comprises n concatenated residual blocks, each of the residual blocks comprises at least one convolutional layer, n is a positive integer greater than 1, and determining the feature maps at the multiple scales of the circuit board image by using the feature extraction network comprises: determining n feature maps of the circuit board image by using the n concatenated residual blocks to serve as the feature maps at the multiple scales, wherein each of the feature maps is an output of the corresponding residual block {see Figs. 4-6 illustrating residual units (residual blocks) convolutional layers (e.g. Conv1 though Conv5) determining feature maps at multiple scales of the circuit board image and including concatenated (Concat) residual blocks as further discussed in section III}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Xin which already teaches determining a first defect region according to a product image including determining feature maps at multiple scales of the product image by using a feature extraction network of a multi-scale detection network; determining fusion features of the feature maps at the multiple scales by using a fusion network of the multi-scale detection network; and determining, according to the fusion features, the first defect region by using a prediction network of the multi-scale detection network such that the conventional details of multi-scale convolutional neural networks are included such as wherein the feature extraction network comprises n concatenated residual blocks, each of the residual blocks comprises at least one convolutional layer, n is a positive integer greater than 1, and determining the feature maps at the multiple scales of the circuit board image by using the feature extraction network comprises: determining n feature maps of the circuit board image by using the n concatenated residual blocks to serve as the feature maps at the multiple scales, wherein each of the feature maps is an output of the corresponding residual block as taught by Hu because Hu teaches that Xin’s image differencing defect detection (corresponding to the determining a second defect region) is conventionally applied to circuit boards, because the product being inspected by Xin also requires careful image registration to accurately determine defects just as circuit board defect detection also requires such careful image registration, because Xin motivates using two different techniques in combination to determine defects, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 4
In regards to claim 4, Xin is not relied upon to disclose but Hu teaches wherein an input layer of the feature extraction network is comprised in a first residual block, and determining the fusion features of the feature maps at the multiple scales by using the fusion network comprises: cascading each of the feature maps into a feature vector through a convolutional operation; upsampling a second feature vector to an nth feature vector to serve as n-1 upsampling vectors; fusing a feature vector corresponding to each residual block with an upsampling vector of a feature vector corresponding to a subsequent residual block, to serve as a fusion feature corresponding to each residual block; and determining a fusion feature of the circuit board image according to the fusion features corresponding to the first residual block to an (n-1)th residual block
{see Figs. 4-6 illustrating residual blocks, cascading via convolution operations, upsampling (series, n-1, of 2x Up Samples) and feature fusion as further discussed in section III}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Xin which already teaches determining a first defect region according to a product image including determining feature maps at multiple scales of the product image by using a feature extraction network of a multi-scale detection network; determining fusion features of the feature maps at the multiple scales by using a fusion network of the multi-scale detection network; and determining, according to the fusion features, the first defect region by using a prediction network of the multi-scale detection network such that the conventional details of multi-scale convolutional neural networks are included such as wherein an input layer of the feature extraction network is comprised in a first residual block, and determining the fusion features of the feature maps at the multiple scales by using the fusion network comprises: cascading each of the feature maps into a feature vector through a convolutional operation; upsampling a second feature vector to an nth feature vector to serve as n-1 upsampling vectors; fusing a feature vector corresponding to each residual block with an upsampling vector of a feature vector corresponding to a subsequent residual block, to serve as a fusion feature corresponding to each residual block; and determining a fusion feature of the circuit board image according to the fusion features corresponding to the first residual block to an (n-1)th residual block as taught by Hu because Hu teaches that Xin’s image differencing defect detection (corresponding to the determining a second defect region) is conventionally applied to circuit boards, because the product being inspected by Xin also requires careful image registration to accurately determine defects just as circuit board defect detection also requires such careful image registration, because Xin motivates using two different techniques in combination to determine defects, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 5
In regards to claim 5, Xin is not relied upon to disclose but Hu teaches wherein determining the fusion feature of the circuit board image according to the fusion features corresponding to the first residual block to the (n-1)th residual block comprises: fusing the fusion features corresponding to the first residual block to the (n-1)th residual block to serve as a first fusion feature; and pooling a region in the first fusion feature to serve as the fusion feature of the circuit board image {see Figs. 4-6 illustrating residual blocks, and ROI pooling to obtain object proposals to serve as the fusion feature of the circuit board image as further discussed in section III}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Xin which already teaches determining a first defect region according to a product image including determining feature maps at multiple scales of the product image by using a feature extraction network of a multi-scale detection network; determining fusion features of the feature maps at the multiple scales by using a fusion network of the multi-scale detection network; and determining, according to the fusion features, the first defect region by using a prediction network of the multi-scale detection network such that the conventional details of multi-scale convolutional neural networks are included such as wherein determining the fusion feature of the circuit board image according to the fusion features corresponding to the first residual block to the (n-1)th residual block comprises: fusing the fusion features corresponding to the first residual block to the (n-1)th residual block to serve as a first fusion feature; and pooling a region in the first fusion feature to serve as the fusion feature of the circuit board image as taught by Hu because Hu teaches that Xin’s image differencing defect detection (corresponding to the determining a second defect region) is conventionally applied to circuit boards, because the product being inspected by Xin also requires careful image registration to accurately determine defects just as circuit board defect detection also requires such careful image registration, because Xin motivates using two different techniques in combination to determine defects, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claims 11-15, and 20
The rejection of method claims 1-5; and 1 above applies mutatis mutandis to the corresponding limitations of device claims 11-15 and computer readable medium claim 20 while noting that the rejection above cites to both device and method disclosures. See also Xin Figs. 1, 3 including processor 104, storage device 106 and software program 108, [0059]-[0081].
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Xin and Hu as applied to claims 1 and 11 above, and further in view of Iwanaga (US 2019/0220972 A1).
Claim 8
In regards to claim 8, Xin is not relied upon to disclose wherein each pixel in the first defect region has a first confidence, each pixel in the second defect region has a second confidence, and determining the defect region of the circuit board according to the first defect region and the second defect region comprises: determining a weighted sum of the first confidence and the second confidence for each pixel according to a first weight and a second weight, to serve as a probability of each pixel belonging to a defect; and determining a set of pixels each having a probability greater than a threshold as the defect region.
Iwanaga is an analogous reference from the same field of circuit board defect detection. See title, [0002]-[0009] and cites below.
Iwanaga also teaches wherein each pixel in the first defect region has a first confidence, each pixel in the second defect region has a second confidence, {See Figs. 3, 4, 9 that determines a first defect region according to the circuit board indicating a location of the defect (first estimation part 4 using deep learning) and having a first confidence (estimation result) and determining a second defect region based on a standard image of the circuit board (second estimation part 6 using rule-based system) indicating a location of the defect and having a second confidence (estimation result), [0032]-[0036]},and
determining the defect region of the circuit board according to the first defect region and the second defect region comprises:
determining a weighted sum of the first confidence and the second confidence for each pixel according to a first weight and a second weight, to serve as a probability of each pixel belonging to a defect {Figs. 3, 4, 9 comprehensive determination part 7 that determines a weighted sum using a priority matrix table for each pixel according to the weights in the priority matrix table to serve as a probability of each pixel belonging to a defect, [0036], [0096]-[102]; and
determining a set of pixels each having a probability greater than a threshold as the defect region {see above and also including [0058]-[0073], [0084]-[0095]}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Xin which already teaches determining defect regions using two different methods (based respectively on a neural network and classical image processing (e.g. template matching) and generally combines the results of the two methods to determine a final defect region such that combining the defect region outputs of these two methods includes wherein each pixel in the first defect region has a first confidence, each pixel in the second defect region has a second confidence, and determining the defect region of the circuit board according to the first defect region and the second defect region comprises: determining a weighted sum of the first confidence and the second confidence for each pixel according to a first weight and a second weight, to serve as a probability of each pixel belonging to a defect, and determining a set of pixels each having a probability greater than a threshold as the defect region as taught by Iwanaga because Xin motivates using two different techniques in combination to determine defects, because Iwanaga motivates using weights (priority matrix) to increase accuracy of the final defect determination, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 18
The rejection of method claim 8 above applies mutatis mutandis to the corresponding limitations of device claim 18 while noting that the rejection above cites to both device and method disclosures. See also Xin Figs. 1, 3 including processor 104, storage device 106 and software program 108, [0059]-[0081].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael R Cammarata whose telephone number is (571)272-0113. The examiner can normally be reached M-Th 7am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ROBERT CAMMARATA/Primary Examiner, Art Unit 2667
Prosecution Insights