DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on March 06, 2025, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the Examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 2021/0334587 A1) in view of Koljonen (US 5,982,927 A).
Regarding claim 1, Wang teaches a solder inspection device comprising:
a position search circuit configured to receive a substrate image of a substrate including a plurality of solder areas and to generate a plurality of search images respectively corresponding to the plurality of solder areas (Fig. 9 shows the method of Wang’s invention. First, a BGA chip—or other similar electronic device [0083]—is imaged. Then, all areas with a solder joint are determined, and a feature image for each solder joint is obtained. [0085] “…a feature image that encloses one feature element therein is obtained associated with a respective one of all feature element regions. Furthermore, the method includes determining an enclosing box of the respective one feature element of the electronic device.” Here, a feature element is a solder joint.);
an inspection image generating circuit configured to generate a plurality of inspection images respectively corresponding to the plurality of solder areas using position information of the plurality of search images (Wang teaches identifying locations of solder joints and determining an enclosing box around each joint. Each enclosing box is the area for each feature image, and the images are subsequently used for machine learning classification of defects using machine learning. [0085] “…a feature image that encloses one feature element therein is obtained associated with a respective one of all feature element regions. Furthermore, the method includes determining an enclosing box of the respective one feature element of the electronic device. [0085] “As shown in FIG. 10, the enclosing box that encloses one feature element 100 in a center region of the image forms a feature image. One initial image of the electronic device may result in multiple feature images.”); and
a classification circuit configured to classify each of the plurality of inspection images as having one of a plurality of states (A CNN receives each feature image and identifies the type of solder defect present. The CNN can also classify solder joints as having no defects. [0086] “Based on the target feature vector, an initial prediction probability of the respective one feature image corresponding to the respective one of different types of defect labels (which are predetermined for the specific feature element of electronic device and summarized as those likely occurred during manufacture process) can be determined from an output of the convolutional neural network (at least from an output of a last layer of classification layer of the CNN). Subsequently, the method includes determining there is no defect in the feature element of the electronic device to qualify the electronic device when none of the initial prediction probability of the respective one feature image corresponding to the respective one of different types of defect labels is greater than a predetermined threshold probability.”).
As shown above, Wang teaches obtaining inspection images by determining the positions of solder areas and marking them using enclosing boxes. Thus, Wang fails to teach an inspection image generating circuit configured to generate a plurality of inspection images respectively corresponding to the plurality of solder areas using a plurality of reference masks.
However, Koljonen teaches an inspection image generating circuit configured to generate a plurality of inspection images respectively corresponding to the plurality of solder areas using a plurality of reference masks (Koljonen teaches using reference masks to determine solder areas and masks them from the background for subsequent defect detection. [Col. 13, lines 8-12] “The difference images (if necessary) or the binary image contains information clearly indicative of the location(s) of solder paste. It is desirable to mask out the pads from the PCB background so that it is possible to separately process pad information and non-pad or background information.”).
Wang and Koljonen are analogous in the art, because both teach methods of inline solder paste inspection. Therefore, it would have been obvious to one of ordinary skill in the art to modify Wang’s invention by utilizing reference masks instead of enclosing boxes. This modification would improve Wang’s invention, since reference masks would more clearly distinguish the solder areas and pads from the irrelevant parts of the circuit board and the background ([Koljonen Col. 13, lines 13-20] “That is, for purposes of further processing and analysis to determine the location of solder on pads and the location of solder off of the pads (likely causing bridges between fine pitch or closely spaced pads), masks are used to select out and independently process pad information and PCB background information. A mask is generated and applied 70 using PCB training information to define on-pad regions 80 and off-pad regions 82 as illustrated in FIG. 5a.”).
Regarding claim 4, Wang and Koljonen teach the solder inspection device of claim 1. Koljonen further teaches wherein the classification circuit is further configured to perform a neural network operation on each of the plurality inspection images and determines each of the plurality of inspection images as having one of a plurality of states based on result of the neural network operation (A CNN receives each feature image and identifies the type of solder defect present. The CNN can also classify solder joints as having no defects. [0086] “Based on the target feature vector, an initial prediction probability of the respective one feature image corresponding to the respective one of different types of defect labels (which are predetermined for the specific feature element of electronic device and summarized as those likely occurred during manufacture process) can be determined from an output of the convolutional neural network (at least from an output of a last layer of classification layer of the CNN). Subsequently, the method includes determining there is no defect in the feature element of the electronic device to qualify the electronic device when none of the initial prediction probability of the respective one feature image corresponding to the respective one of different types of defect labels is greater than a predetermined threshold probability.”).
Claims 2-3 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 2021/0334587 A1) and Koljonen (US 5,982,927 A), further in view of Ti et al. (CN 109636787 B), hereafter Ti.
Regarding claim 2, Wang and Koljonen teach the solder inspection device of claim 1. Wang further teaches to extract the plurality of inspection images from the substrate image using the plurality of reference masks (Each enclosing box is the area for each feature image. [0085] “…a feature image that encloses one feature element therein is obtained associated with a respective one of all feature element regions. Furthermore, the method includes determining an enclosing box of the respective one feature element of the electronic device. [0085] “As shown in FIG. 10, the enclosing box that encloses one feature element 100 in a center region of the image forms a feature image. One initial image of the electronic device may result in multiple feature images.”).
Wang teaches utilizing enclosing boxes instead of reference masks; thus, Wang fails to teach extracting the plurality of inspection images from the substrate image using the plurality of reference masks. However, Koljonen teaches extracting the plurality of inspection images from the substrate image using the plurality of reference masks ([Col. 13, lines 8-12] “The difference images (if necessary) or the binary image contains information clearly indicative of the location(s) of solder paste. It is desirable to mask out the pads from the PCB background so that it is possible to separately process pad information and non-pad or background information.”).
As discussed above in the rejection to claim 1, It would be obvious to one of ordinary skill in the art to modify Wang’s invention to utilize reference masks in place of enclosing boxes for marking solder areas for inspection images. This modification would allow for solder areas and pads to be better distinguished from the background and irrelevant parts of the PCB under inspection ([Koljonen Col. 13, lines 13-20] “That is, for purposes of further processing and analysis to determine the location of solder on pads and the location of solder off of the pads (likely causing bridges between fine pitch or closely spaced pads), masks are used to select out and independently process pad information and PCB background information. A mask is generated and applied 70 using PCB training information to define on-pad regions 80 and off-pad regions 82 as illustrated in FIG. 5a.”).
Additionally, Wang and Koljonen fail to teach wherein the inspection image generating circuit is further configured to adjust positions of the plurality of reference masks based on position information of the plurality of search images.
However, Ti teaches wherein the inspection image generating circuit is further configured to adjust positions of the plurality of reference masks based on position information of the plurality of search images (See Embodiment 9 of Ti described with reference to Fig. 1 at [0071-0079]. Ti teaches determining solder areas on a battery under inspection, marking each solder area with a bounding box (reference masks would be obvious substitutes for bounding boxes in view of Koljonen discussed above), calculating the average center position of the bounding boxes, and repositioning the bounding boxes by aligning their average center coordinate with the center of the candidate solder joint image.).
Wang, Koljonen, and Ti are all analogous to the claimed invention, because all teach identifying solder areas on an object under inspection and determining if defects are present in the solder areas. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention taught by Wang and Koljonen by allowing for positional adjustments of the plurality of bounding boxes or reference masks. This modification would allow for the average center of the bounding boxes or reference masks to be aligned with the center of the inspection image(s) (See [0071-0079] of Ti discussing the process of aligning the bounding boxes for alignment with image(s).) Furthermore, Ti explains that this correction is necessary for adjusting the size of the image(s) to be a standard size. The correction would allow for proper alignment of the bounding boxes to the images after resizing ([Ti 0074] “…step three, adjusting the size of the binary image of the candidate welding spot to the standard size. Obtaining candidate welding spot image I cddt_point Firstly, finding out the minimum bounding rectangle of all candidate welding point areas and the central position of the minimum bounding rectangle, Calculating the average center position of all rectangles; Translating to candidate welding point image I cddt_point.”).
Regarding claim 3, Wang, Koljonen, and Ti teach the solder inspection device of claim 2. Ti further teaches wherein the inspection image generating circuit is further configured to adjust the positions of the plurality of reference masks using a first coordinate representing an average of center coordinates of the plurality of reference masks and a second coordinate representing an average of center coordinates of the plurality of search images (See Embodiment 9 described with reference to Fig. 1 at [0071-0079]. [0074] “…step three, adjusting the size of the binary image of the candidate welding spot to the standard size. Obtaining candidate welding spot image I cddt_point Firstly, finding out the minimum bounding rectangle of all candidate welding point areas and the central position of the minimum bounding rectangle, Calculating the average center position of all rectangles; Translating to candidate welding point image I cddt_point.” Ti teaches adjusting the bounding box locations (reference masks would be obvious substitutes for bounding boxes in view of Koljonen discussed above in claims 1-2) to align with the welding point image(s) by using both the coordinate representing of the average of center coordinates of the bounding boxes and the coordinate representing of the average of center coordinates of the welding point image(s).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention taught by Wang and Koljonen by allowing for positional adjustments of the plurality of bounding boxes or reference masks. This modification would allow for the average center of the bounding boxes or reference masks to be aligned with the center of the inspection image(s) (See [0071-0079] of Ti discussing the process of aligning the bounding boxes for alignment with image(s).) Furthermore, Ti explains that this correction is necessary for adjusting the size of the image(s) to be a standard size. The correction would allow for proper alignment of the bounding boxes to the images after resizing ([Ti 0074] “…step three, adjusting the size of the binary image of the candidate welding spot to the standard size. Obtaining candidate welding spot image I cddt_point Firstly, finding out the minimum bounding rectangle of all candidate welding point areas and the central position of the minimum bounding rectangle, Calculating the average center position of all rectangles; Translating to candidate welding point image I cddt_point.”).
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 2021/0334587 A1) and Chen et al. (CN 112990326 A), hereafter Chen.
Regarding claim 5, Wang teaches a method of operating a solder inspection device, the method comprising: receiving, by a classification neural network of the solder inspection device, a plurality of inspection images of solder areas of a substrate (Wang teaches identifying locations of solder joints and determining an enclosing box around each joint. Each enclosing box is the area for each feature image. [0085] “…a feature image that encloses one feature element therein is obtained associated with a respective one of all feature element regions. Furthermore, the method includes determining an enclosing box of the respective one feature element of the electronic device. [0085] “As shown in FIG. 10, the enclosing box that encloses one feature element 100 in a center region of the image forms a feature image. One initial image of the electronic device may result in multiple feature images.”);
assigning one of a first state and a second state to the inspection image (See Fig. 5. The CNN determines whether there is a defect (first state) in an inspection image or whether there is no defect (second state) in an inspection image. Wang also teaches supervised learning, so the inspection images are labeled with the ground truth or target value before training.);
determining, in a first epoch learning of the classification neural network, a number of first cases for which a sample of the first inspection image having the first state is classified as having the first state, and a number of second cases for which the sample of the first inspection image having the first state is classified as having the second state ([0044-0052] and Fig. 1 are directed to the overall training method for the CNN. The CNN receives inspection images as input, and the CNN outputs a defect label for each image. As seen in Fig. 5, the results determine whether a defect is present or not in each image, and network parameters are updated based on the results. Considering that Wang teaches supervised learning, one of ordinary skill in the art would be expected to be able to obtain the results of the training, which would include the number of images containing defects which were correctly labeled (first case) and the number of images containing defects which were incorrectly labeled as defectless (second case).).
Wang teaches selecting the samples which are included in the training set. In [0047], Wang explains that the training set should include “a proper number” of solder joint images containing one or more defects, but Wang does not teach adjusting a first selection probability that a sample of the first state is selected and a second selection probability that a sample of the second state is selected during the next epoch learning.
However, Chen teaches adjusting a first selection probability that a sample of the first state is selected and a second selection probability that a sample of the second state is selected during the next epoch learning (At each epoch, the sampling probability per a class is adjusted based on the recall. [0009] “Before the start of each training round, the performance of the imbalanced data classification network is tested on the validation set to obtain the classification recall for each category. The sampling weight for each category is obtained based on the classification recall, and the sampling probability of the sample is obtained based on the sampling weight.”).
Wang and Chen are analogous in the art, because both teach methods for training a classification model. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wang’s invention by applying Chen’s teachings to handle imbalanced training data. Chen teaches a general method which could be applied to any type of classifier—such as the defect detection classifier taught by Wang—and one would be motivated to apply this method to increase the accuracy of Wang’s classifier due to imbalanced training data. In [0047], Wang motivates handling imbalanced data by explaining that the “proper number” of samples and a variety of samples among classes are both necessary for better training of the CNN. Furthermore, Chen teaches that adjusting the amount of samples from each class in the training data based on the model performance at each epoch will improve the model performance ([Chen 0031] “The beneficial effects of this invention are: This invention dynamically adjusts the sampling probability based on the performance of the imbalanced data classification network on the validation set, so that the number of samples in different categories changes with the optimization needs of the imbalanced data classification network; if the imbalanced data classification network performs well in a certain category, the number of samples in that category is reduced, so that the model pays more attention to samples in other categories. Furthermore, the method proposed in this invention fully utilizes the difference between the training set and the validation set samples. During training, the performance of the imbalanced data classification network on the validation set is used to guide its training on the training set, thereby further improving the generalization performance of the imbalanced data classification network.”).
Regarding claim 6, Wang and Chen teach the method of claim 5. Chen further teaches wherein adjusting the first selection probability and the second selection probability comprises: decreasing the first selection probability and increasing the second selection probability when the number of first cases is smaller than the number of second cases ([0047] “Before the start of each training round, test the performance of the imbalanced data classification network on the validation set, obtain the classification recall for each category, obtain the sampling weight for each category based on the classification recall, and obtain the sampling probability of the sample based on the sampling weight.” Here, Chen teaches that the sampling probability for a class is adjusted based on the recall for that class. Regarding claim 6 of the claimed invention, the first state (or class) is images which contain a defect. The recall for the first state would consider the ratio of the first cases (images containing defects which were classified correctly) to the second cases (images containing defects which were classified incorrectly).).
Therefore, it would have been obvious to one of ordinary skill in the art to modify Wang’s invention by applying Chen’s teachings to handle imbalanced training data. Chen teaches a general method which could be applied to any type of classifier—such as the defect detection classifier taught by Wang—and one would be motivated to apply this method to increase the accuracy of Wang’s classifier due to imbalanced training data. In [0047], Wang explains that the “proper number” of samples and a variety of samples among classes is necessary for better training of the CNN. Furthermore, Chen teaches that adjusting the amount of samples from each class in the training data based on the model performance at each epoch will improve the model performance ([Chen 0031] “…if the imbalanced data classification network performs well in a certain category, the number of samples in that category is reduced, so that the model pays more attention to samples in other categories.”).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 2021/0334587 A1) and Chen (CN 112990326 A), and further in view of Zeng et al. (CN 112967243 B), hereafter Zeng.
Regarding claim 9, Wang and Chen teach the method of claim 5. Wang further teaches classifying many different types of solder defects ([0004] “the method includes adjusting network parameters characterizing the convolutional neural network through a training loss function associated with a classification layer based on the target feature vectors and pre-labeled defect labels corresponding to different types of solder joint defects.”). However, Wang does not specifically teach wherein the first state is one of a solder crack occurrence, a printed circuit board (PCB) crack occurrence, and a package crack occurrence, and the second state is one of a solder crack non-occurrence, a PCB crack non-occurrence, and a package crack non-occurrence.
However, Zeng teaches wherein the first state is one of a solder crack occurrence, a printed circuit board (PCB) crack occurrence, and a package crack occurrence, and the second state is one of a solder crack non-occurrence, a PCB crack non-occurrence, and a package crack non-occurrence (Zeng teaches training a neural network to detect packaging crack defects. [0010-0016] “The main objective of this invention is to overcome the shortcomings of the aforementioned background technology and provide a method for detecting crack defects in deep learning chip packaging based on YOLO. To achieve the above objectives, the present invention adopts the following technical solution: A YOLO-based deep learning chip packaging crack defect detection method includes the following steps: The first step is to acquire images of the chip units… The fourth step is to construct a deep learning network model for defect detection based on the YOLOv4 network.”).
Wang, Chen, and Zeng are analogous in the art to the claimed invention, because all teach methods of training a machine learning model. Furthermore, Wang and Zeng are directly analogous to the claimed invention, because both teach utilizing machine learning specifically for classifying defects on circuit boards and/or chips. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention taught by Wang and Chen by extending the types of defects to include cracking. Wang’s invention teaches a defect detection system which can be extended to more types of defects, and Zeng teaches that package cracking is a relevant and important type of defect to detect with machine learning systems ([0002-0005] “The main defects appear on the surface of the chip, and different types of defects are accompanied by different abnormal appearance features such as cracks. An important part of the packaging test is to detect defects on the chip surface. Chip packaging and testing is a crucial step in eliminating defective products and ensuring product quality and production reliability. Emerging fields such as artificial intelligence, the Internet of Things, and cloud computing all rely on advanced packaging and testing technologies… Deep learning-based methods leverage the powerful image feature extraction capabilities of convolutional neural networks to learn from training samples containing different types of chip defects.”).
Allowable Subject Matter
Claims 7-8 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 7, Wang and Chen teach the method of claim 5, but Wang and Chen fail to teach wherein a value of a loss function when a sample is applied to the classification neural network includes a first term corresponding to the first case and a second term corresponding to the second case, the method further comprising: determining the first term by multiplying a first loss parameter and a first cross-entropy function; and determining the second term by multiplying a second loss parameter and a second cross-entropy function.
Wang teaches updating model parameters and the loss function(s) based on target feature vectors and pre-labeled defect labels for different types of defects ([0004] “Additionally, the method includes adjusting network parameters characterizing the convolutional neural network through a training loss function associated with a classification layer based on the target feature vectors and pre-labeled defect labels corresponding to different types of solder joint defects.”). A Sigmoid cross-entropy loss function is updated by comparing the machine learning prediction to the target prediction ŷm and the ground truth prediction ym for a sample of a class. See [0007] for the loss formula and explanation of terms. Wang does not teach or motivate utilizing terms specific to the first case and the second case as required by claim 7 of the claimed invention.
As shown above in the rejection to claim 5, Chen teaches methods for handling imbalanced data sets as training progresses by considering the first cases and second cases for updating prediction probabilities. Furthermore, Chen teaches utilizing a cross-entropy loss function in some embodiments. However, Chen does not teach including parameters which correspond to the first cases and the second cases in the loss function. Rather, Chen teaches updating the machine learning parameters and the loss function based on the prediction accuracy as conventionally performed in supervised learning tasks. The recall (which considers the first and second cases), is used to update the prediction probabilities for selecting samples of a certain class for the next epoch.
Therefore, both Wang and Chen fail to teach each and every limitation of claim 7.
Regarding claim 8, this claim is dependent upon claim 7, so it would be allowable for the same reasons as claim 7 if rewritten in independent form including all of the limitations of the claims it depends upon, which includes both claims 5 and 7.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Xia et al. (US 2020/0292471 A1) teaches methods for utilizing machine learning for inspecting the quality of solder paste on printed circuit boards.
Kim et al. (US 2015/0210064 A1) teaches methods for correcting the position of a mask receiving fiducial information, determining the positions of solder on a board, and translating the mask based on fiducials and solder locations. Kim et al. also teaches methods for inspecting solder quality.
Mirzaei (Automating Fault Detection and Quality Control in PCBs: A Machine Learning Approach to Handle Imbalanced Data. Masters thesis, Concordia University. Available online: https://spectrum.library.concordia.ca/id/eprint/992953/) teaches methods for utilizing machine learning to classify solder defects, and Mirzaei teaches methods for handling imbalanced data sets.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC JAMES SHOEMAKER whose telephone number is (571)272-6605. The examiner can normally be reached Monday through Friday from 8am to 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner' s supervisor, JENNIFER MEHMOOD, can be reached at (571)272-2976. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Eric Shoemaker/
Patent Examiner
/JENNIFER MEHMOOD/ Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights