Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 5 and 6 have been amended.
Claims 1-10 and 13-18 are still pending for consideration.
The rejection of claims 5 and 6 under 35 U.S.C 112(b) as being indefinite have been withdrawn following Applicant’s amendment.
Response to Arguments
Applicant's arguments filed on 08/17/2026 have been fully considered but they are not persuasive.
On page 5 of the “Remark”, applicant asserts “Independent claim 1 requires preparing first teaching data using defective original image data and marked image data and preparing second teaching data using non- defective original image data and identity mapping data having no difference from the original image data. Claim 1 further requires preparing a learned machine learning model by performing learning using both the first teaching data and the second teaching data. Thus, the same claimed image-input/image-output learning framework learns both defect- to-predetermined-color mapping and non-defective identity mapping.
Banno does not teach this arrangement…….. Banno therefore teaches image-to- classification learning, not learning of an image-to-image correspondence between original image data and marked image data. For the same reason, Banno's non-defect label is not the claimed identity mapping data. An "OK" or other non-defect classification label is not image data having no difference from the original image data. Nor does assigning such a classification label cause the identifier to learn a correspondence in which the output image is the same image as the input image…. Namiki does not cure these deficiencies…… Rothmund likewise does not teach the claimed common hybrid model”.
Applicant further on page 6 asserts “The asserted combination therefore would require more than adding Rothmund's defect-free training images to Banno and Namiki. It would require changing the Banno/Namiki classification architecture into an image-input/image-output architecture, replacing classification labels with marked image teaching outputs, incorporating identity- mapping image outputs for non-defective training, and training these mappings in a common model”.
Response:
Applicant’s arguments are not persuasive when the references are considered for their collective teachings.
With respect to Banno, the Examiner does not rely upon Banno’s “OK” classification label itself as constituting the claimed “identity mapping data having no difference from the original image data.” Rather, Banno is relied upon for generating teaching data from detect-containing and non-defect-containing image information, tiled learning, overlapping adjacent image segments, and treatment of tiled segments lacking defect information as a non-defective teaching example.
Banno teaches generating teaching data from original inspection images including defective images. Banno explains that “The training data group TrD contains OK image Im1 and NG image Imd (Im2 and Im3) shown in FIG. 3” (see para [0064]), thereby providing both non-defective and defective inspection image information for generating teaching data. For the defective image, Bonno teaches on Abstract “A second pixel value differing from a first pixel value, which is assigned to a defect-free region, is assigned to a region corresponding to a defect region… divided into multiple image segments. A defect label is assigned to the image segment in which the second pixel value …and a non-defect label is assigned to the image segment”. Banno additionally expressly teaches the claimed “adjacent image segments are partially overlapped with each other”. Banno then uses the resulting segmented data for machine learning, expressly teaching that; “the learning unit 64 performs supervised learning using the labeled image segment data DIm as teaching data” (see para [0110]). Significantly, Banno also expressly addresses the situation in which a tiled segment originates from a defective/NG image, but the particular segment does not contain the defect. Banno on para [0106] disclose “In step S27, the computer 32 sets the label of the image segment DI to “OK”. That is, even when the image segment DI is not an OK image (or NG image) and includes the α-channel, if the defect region DA is not included in the image segment DI, the computer 32 sets the label of the image segment DI to “OK””. Banno characterizes this operation as: “assigning a non-defect label” to an image segment in which the defect-indicating second pixel value is absent.
Thus, Banno expressly teaches that a tiled segment originating from defect-containing image data, but lacking the defect indication in that particular segment, is treated as a non-defective teaching example. This directly supports the claimed treatment of tiled teaching data having no marking portion.
Applicant’s observation that Banno ultimately employs classification labels does not negate these teachings because the rejection does not rely upon Banno alone for the particular form of the claimed non-defective output image.
Namiki supplies additional teachings concerning the form of the training image data. For a non-defective/OK condition, Namiki expressly teaches creating training data using: “a training image which is the image as inputted”, Namiki further explains the same principle by teaching that: “the image captured and inputted itself is used as is as the training image” (see para [0066]). Accordingly, Namiki expressly teaches preserving the original captured image itself without modification for the non-defective training condition. Namiki also teaches “the symbol 22 is marked to overlap the defect 20” and “a predetermined color may be given so as to surround the defect 20” (see para [0060]). Namiki additionally links the defect indication to the type of defect, teaching that: “the symbol indicating the defect may be a different symbol depending on a type of the defect”.
Accordingly, Namiki teaches two relevant training conditions: an unchanged original image for the non-defective/OK condition and a defect-marked image for defective condition, with the defect indication capable of being provided using a predetermined color and varied according to defect type.
Rothmund supplies complementary image-to-image reconstructing teaching. Rothmund trains defect-free images so that reconstructed image portions are “as close to the original as possible” and compares “a structural similarity between the reconstructed image and the original image” (see para [0114]). Rothmund further explains that, where no anomaly is present, the received image and reconstructed image are “then the received…and the reconstructed image are essentially the same”. Thus, Rothmund teaches reconstruction of defect-free image information toward the corresponding original image.
Applicant’s argument that Rothmund requires separate networks is also unpersuasive. Ruthmund expressly teaches a training set comprising “both a plurality of training images each showing a defective medical product and a plurality of training images each showing a defect-free medical product…… a defect detection by the defect detector becomes more reliable and more accurate”.
Accordingly, one of ordinary skill in the art would have been motivated to combine Banno’s tiled detect/non-defect training. Namiki’s unchanged non-defective images and defect-type-dependent markings, and Rothmund’s reconstruction toward the original image and common defective/non-defective training. The combination would predictably provide non-defective teaching data having no intended difference from the original image while providing defect-indicating marked teaching data for defective images, with Rothmud expressly supplying improved reliability and accuracy as a reason for using both defective and defect-free examples.
Therefore, Applicant’s arguments do not overcome the collective teaching of Banno, Namiki, and Rothmund, and the rejection of claim is maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-2, 13, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Banno et al. (US 20210166374 A1) in view of Namiki (US 20190012579 A1) and further in view of Rothmund et al. (US 20230022631 A1).
Regarding claim 1, Banno et al. teaches a defect detection method comprising steps of: applying, by a processing circuit, (see para [0050]; “The image data of the item 12 captured by the camera 31 is sent to the computer 32. The computer 32 inspects the item 12 for defects using the image data sent from the camera 31…. the computer 32 may be a microprocessor”), a predetermined color for each type of defect to a defective portion having a contour shape and a defective portion having a region shape that may be present in an inspection target object in an original image obtained by imaging of the inspection target object (see para [0076]; “along the contour of the defect Dg to outline a defect region DA…the channel addition section 61A adds the alpha channel Imα to the NG image data Imd. Then, the pixel value assignment section 61B assigns a second pixel value P2 as the pixel value that indicates a defect to a region Da (hereafter, also referred to as “defect region Da”) corresponding to the defect region DA in the alpha channel”, see also para [0012]; “a different second pixel value is assigned in accordance with the type of the defect” and see para [0095]; “assigned to the region corresponding to the defect region in the other channel of the image as the pixel value indicating a defect” Note: outlining a contour, defining a region (DA), and assigning specific pixel values to represent the defect type in an image channel); preparing one or more sets of first teaching data by the processing circuit using as the first teaching data, original image data obtained by imaging of the inspection target object and having a defect (see para [0064]; “the image data group IG includes a training data group TrD and a test data group TsD. The image data Im belonging to the training data group TrD serves as the original image data when the teaching data generation unit 52 generates teaching data for supervised learning. The training data group TrD contains OK image Im1 and NG image Imd (Im2 and Im3) shown in FIG. 3”, see also para [0074]; “The image data Im is the image data Imd of a defective item (also referred to as “NG image data Imd”) including a defect Dg”); wherein, in the step of preparing a learned machine learning model, tiling processing is performed on the original image data and the marked image
data in the first teaching data to obtain segmented images (see para [0006]; “In machine learning such as deep learning, a massive number of images need to be learned to determine a parameter of an identifier. For example, when an inspection is performed using an identifier, precision is improved if the image is divided .. In this case, a label needs to be assigned to each image segment”, see also para [0010]; “In the segmentation step, the image including the other channel is segmented into multiple image segments”, see also para [0014]; “segmented into the multiple image segments in which adjacent image segments are partially overlapped with each other”), and then learning of a
correspondence relationship between the original image data and the marked image
data is performed for each tiled segmented image (see para [0020]; “an identifier is generated by performing supervised or semi-supervised machine learning using labeled image data as teaching data, the labeled image data being generated by the method for generating labeled image data. In the determination step, multiple image segments are obtained by segmenting an image of an inspection subject into the number of segments that is the same as the number of segments of the image segments used for generating the identifier”),
in both types of the tiling processing, segmented images are created such that,
after tiling, both edge portions of adjacent segmented images overlap (see para [0100]; “the image Im is divided into M×N of the image segments DI having the same size and overlapped with the adjacent image segments DI by the overlap amount ΔL or 2.Math.ΔL”), and further, in a case where a marked image data as a segmented image in the first teaching data does not include a marking portion, learning of a correspondence relationship
between original image data as the segmented image and identity mapping data of the original image data is performed (see para [0106]; “step S27 correspond to a process for assigning a non-defect label to the image segment DI, in which the second pixel value P2 indicating a defect is not included in the other channel Therefore, the processes of steps S25 to S27 corresponds to one example of “label assignment step” Note; learning the correspondence between a normal original image and its corresponding normal mask (identity mapping, where input = output, mapping defect-free to defect-free) is fundamentally the same as assigning a "non-defect" label to an image segment. Therefore, steps S25–S27, which handle this logic, act as a label assignment step to identify that an image segment is healthy or non-defective). However, Bunno et al. does not teach and marked image data obtained by labeling with a predetermined color for each type of defect on the original image data in the step of applying a predetermined color, preparing one or more sets of second teaching data by the processing circuit using, as the second teaching data, original image data obtained by imaging of the inspection target object and having no defect, and identity mapping data having no difference from the original image data; and preparing a learned machine learning model by the processing circuit performing learning using the first teaching data and the second teaching data.
In the same field of endeavor, Namiki teaches and marked image data obtained by labeling with a predetermined color for each type of defect on the original image data in the step of applying a predetermined color (see para [0060]; “a predetermined color may be given so as to surround the defect 20 on the surface of the inspection target 12 by a predetermined color pen”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al. in view of a machine learning device that creates training data to be used in machine learning of Namiki in order to distinguish between OK and NG on new input images of the inspection target (see para [0060]).
However, the combination of Bunno et al. Namiki as a whole does not teach preparing one or more sets of second teaching data by the processing circuit using, as the second teaching data, original image data obtained by imaging of the inspection target object and having no defect, and identity mapping data having no difference from the original image data; and preparing a learned machine learning model by the processing circuit performing learning using the first teaching data and the second teaching data.
In the same field of endeavor, Rothmund et al. teaches preparing one or more sets of second teaching data by the processing circuit using, as the second teaching data, original image data obtained by imaging of the inspection target object and having no defect (see para [0005]; “first neural network trained based on a first set of training images, and the first set of training images comprises a plurality of training images each showing a corresponding defect-free product…. one third set of training images, and the at least one third set of training images comprises a plurality of training images each showing a corresponding defective product”) and identity mapping data having no difference from the original image data (see para [0073]; “the at least one first neural network is trained to reconstruct the patches at the edge regions to be as similar as possible to the original image when training the same”); and preparing a learned machine learning model by the processing circuit performing learning using the first teaching data and the second teaching data (see para [0006]; “the first set of training images comprising a plurality of training images each showing a defect-free product; training at least a first neural network of an inpainting autoencoder based on the first set of training images….. creating at least one third set of training images, wherein the at least one third set of training images comprises a plurality of training images each showing a defective product, and further at least one defect is marked in each training image of the third set of training images; training at least one third neural network of a defect identifier based on the at least one third set of training images”), tiling processing is similarly performed on the original image data and the identity mapping data in the second teaching data to obtain segmented images, and then learning of a correspondence relationship between the original image data and the identity mapping data is performed for each tiled segmented image (see para [0005]; “wherein the autoencoder includes at least one first neural network trained based on a first set of training images, and the first set of training images includes a plurality of training images each showing a corresponding defect-free product”, see also para [0097]; “perform the reconstruction of the individual patches in parallel. The reconstructed patches can then be reassembled to form a reconstructed overall image”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al. in view of a machine learning device that creates training data to be used in machine learning of Namiki and further in view of a method of analyzing a product includes performing an anomaly detection on a received image using an autoencoder of Rothmund et al. in order to generate anomaly-free equivalent of the received image with the same position or orientation (see para [0005]).
Regarding claim 2, the rejection of claim 1 is incorporated herein.
Namiki in the combination further teach wherein the machine learning model includes a convolutional neural network (CNN) or a fully convolutional neural network (FCNN) (see para [0054]; “the learning model constructed by a neural network including an input layer, output layer and intermediate layer can use an appropriate system. For example, CNN (Convolutional Neural Network) can also be applied”).
Regarding claim 13, the rejection of claim 1 is incorporated herein.
Namiki in the combination further teach a storage medium storing a computer program causing the processing circuit to execute the defect detection method (see para [0007]; “computer program product storing instructions embodied on a computer-readable medium or programmable circuitry, that, when executed by a processor or the programmable circuitry, cause the processor or the programmable circuitry to perform the method”).
Regarding claim 16, the rejection of claim 1 is incorporated herein.
Namiki in the combination further teach a defect detection system comprising a computer device and a storage device, wherein the computer device performs the defect detection method (see para [0007]; “computer program product storing instructions embodied on a computer-readable medium or programmable circuitry, that, when executed by a processor or the programmable circuitry, cause the processor or the programmable circuitry to perform the method”).
Claims 3, 14, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Banno et al., and Namiki et al. in view of Rothmund et al as applied in claim 1 above, and further in view of Kimura et al. (US 20200104990 A1) and Majurski et al. NPL “Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory”.
Regarding claim 3, the rejection of claim 1 is incorporated herein.
Banno et al. in the combination further teach by inputting image data of the inspection target object to the machine learning model (see para [0028]; “The identifier is generated by performing supervised or semi-supervised machine learning using the labeled image data, which is generated by the labeled image data generation device, as teaching data…..The determination unit determines whether the inspection subject is defective based on an output result from the identifier when the image segments obtained by the image segment acquisition unit is input to the identifier”), the input image data having been subjected to the tiling processing (see para [0054]; “The image segment data is obtained by dividing the image of the item 12 captured by the camera 31”). However, the combination of Bunno et al., Rothmund et al and Namiki as a whole does not teach acquiring predicted image data by the processing circuit subtracting input image data from output image data.
In the same field of endeavor, Kimura et al. teaches acquiring predicted image data by the processing circuit subtracting input image data from output image data (see para [0063]; “The calculating section may subtract the reconstructed dataset from the original dataset to generate a differential dataset as the difference between the original and reconstructed datasets”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al. in view of a machine learning device that creates training data to be used in machine learning of Namiki and further in view of a method of analyzing a product includes performing an anomaly detection on a received image using an autoencoder of Rothmund et al. and detecting an anomaly by using the weighted differential dataset of Kimura et al. in order to improve accuracy of anomaly detection (see para [0063]). However, the combination of Bunno et al., Rothmund et al, Namiki and Kimura et al. as a whole does not teach and performing reverse tiling processing, by the processing circuit, on the predicted image data, the reverse tiling processing performing combination by ignoring a portion of an outer half of an overlapping edge portion in each piece of the predicted image data before combination.
In the same field of endeavor, Majurski et al. teaches and performing reverse tiling processing, by the processing circuit, on the predicted image data, the reverse tiling processing performing combination by ignoring a portion of an outer half of an overlapping edge portion in each piece of the predicted image data before combination (see Abstract; “Our approach is to select a tile size that will fit into GPU memory with a halo border of half the network receptive field…..Next, stride across the image by that tile size without the halo. The input tile halos will overlap, while the output tiles join exactly at the seams”, see also page 5, 1st para; “The ZoR needs to be cropped out from the prediction so only those pixels with full context are used to build the final result”, Note; recombination of tiled outputs by excluding overlap-edge regions before stitching corresponds to “performing combination by ignoring a portion of an outer half of an overlapping edge portion in each piece of the predicted image data before combination”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al. in view of a machine learning device that creates training data to be used in machine learning of Namiki and further in view of a method of analyzing a product includes performing an anomaly detection on a received image using an autoencoder of Rothmund et al. and detecting an anomaly by using the weighted differential dataset of Kimura et al. and further in view of exact tile-based segmentation inference for images larger than GPU memory of Majurski et al. in order to estimate the tiling parameter (see para [0063]).
Regarding claim 14, the rejection of claim 3 is incorporated herein.
Namiki in the combination further teach a storage medium storing a computer program causing the processing circuit to execute the defect detection method (see para [0007]; “computer program product storing instructions embodied on a computer-readable medium or programmable circuitry, that, when executed by a processor or the programmable circuitry, cause the processor or the programmable circuitry to perform the method”).
Regarding claim 17, the rejection of claim 3 is incorporated herein.
Namiki in the combination further teach a defect detection system comprising a computer device and a storage device, wherein the computer device performs the defect detection method (see para [0007]; “computer program product storing instructions embodied on a computer-readable medium or programmable circuitry, that, when executed by a processor or the programmable circuitry, cause the processor or the programmable circuitry to perform the method”).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Banno et al., Namiki and Rothmund et al. in view Kimura et al. and Majurski et al. of as applied in claims 1 and 3 above, and further in view of Hou et al. (US 20130223758 A1).
Regarding claim 4, the rejection of claim 3 is incorporated herein. The combination of Bunno et al., Rothmund et al, Namiki, Kimura et al. and Majurski et al. as a whole does not teach wherein in the step of performing the reverse tiling processing, hue, saturation, and brightness are adjusted based on a predetermined color for each type of defect between a plurality of pieces of the predicted image data before combination.
In the same field of endeavor, Hou et al. teaches wherein in the step of performing the reverse tiling processing, hue, saturation, and brightness are adjusted based on a predetermined color for each type of defect between a plurality of pieces of the predicted image data before combination (see para [0036]; “the categorization is undertaken by analyzing the brightness and saturation of pixels…. the brightness value (1 component in the HIS color model) and saturation value (S component in the HIS colour model)”, see also para [0044]; “The new HIS value of P0 is P0+DIFF/2, and the new HIS value of its corresponding pixel P1 is P1-DIFF/2”, and para [0057]; “The blended image portions are combined”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al., Namiki and Rothmund et al. in view of detecting an anomaly by using the weighted differential dataset of Kimura et al. and exact tile-based segmentation inference for images larger than GPU memory of Majurski et al. and further in view of a method of blending stitched document image portions of Hou et al. in order to reduce visually perceivable seams (see para [0036]).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Banno et al., Namiki et al., and Rothmund et al in view of Kimura et al., Majurski et al. and Hou et al. as applied in claims 1, 3 and 4 above, and further in view of Matsumura (US 20060078191 A1).
Regarding claim 5, the rejection of claim 4 is incorporated herein.
Hou et al. in the combination further teach further comprising steps of: obtaining, by the processing circuit, as detection image data, the predicted image data after the reverse tiling processing; and classifying, by the processing circuit, the defect based on a correspondence between the predetermined colors and the respective types of defects and on a hue of the portion (see para [0036]; “the categorization is undertaken by analyzing the brightness and saturation of pixels…. the brightness value (1 component in the HIS color model) and saturation value (S component in the HIS colour model)”, see also para [0044]; “The new HIS value of P0 is P0+DIFF/2, and the new HIS value of its corresponding pixel P1 is P1-DIFF/2”, and para [0057]; “The blended image portions are combined”).
The combination of Bunno et al., Rothmund et al, Namiki, Kimura et al. Majurski et al. and Hou et al. as a whole does not teach obtaining, by the processing circuit, an average value and a standard deviation of pixel values of all pixels of the detection image data; determining, by the processing circuit, a detection threshold based on the standard deviation; identifying, by the processing circuit, pixels each having a pixel value for which an absolute difference from the average value is at least the detection threshold; detecting, by the processing circuit, a portion including the identified pixels as a defect.
In the same field of endeavor, Matsumura teach obtaining, by the processing circuit, an average value and a standard deviation of pixel values of all pixels of the detection image data (see para [0057]; “if an absolute value of difference between a value and an average value of the values of pixels therein is larger than three times a standard deviation”); determining, by the processing circuit, a detection threshold based on the standard deviation (see para [0057]; “the defect candidate pixel threshold value is determined to be 3”, see also para [0059]; “the differential image is compared has only to be substantially based on a standard deviation”); identifying, by the processing circuit, pixels each having a pixel value for which an absolute difference from the average value is at least the detection threshold (see para [0057]; “if an absolute value of difference between a value and an average value of the values of pixels therein is larger than three times a standard deviation”); detecting, by the processing circuit, a portion including the identified pixels as a defect (see para [0059]; “this group of pixels is regarded as a defect candidate”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al., Namiki and Rothmund et al. in view of detecting an anomaly by using the weighted differential dataset of Kimura et al. and exact tile-based segmentation inference for images larger than GPU memory of Majurski et al. and further in view of a method of blending stitched document image portions of Hou et al. and apparatus and method for detecting defect on object of Matsumura in order to detect a defect on a substrate with high accuracy and high efficiency (see para [0050]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Banno et al., Namiki et al., Rothmund et al in view of Kimura et al., Majurski et al. Hou et al. and Matsumura as applied in claims 1, 3 and 4 above, and further in view of Hyatt (US 20210233229 A1).
Regarding claim 6, the rejection of claim 4 is incorporated herein.
Kimura et al. in the combination further teach and obtained by subtracting the input image data from the output image data (see para [0063]; “The calculating section may subtract the reconstructed dataset from the original dataset to generate a differential dataset as the difference between the original and reconstructed datasets”).
Matsumura in the combination further teach further comprising steps of: :comparing, by the processing circuit, a color of a portion included in the predicted image data with the predetermined colors corresponding to the respective types of defects; in response to determining that the color of the portion does not match any of the predetermined colors, calculating, by the processing circuit, a cumulative difference value for the portion (see para [0052]; “a group of pixels which are connected with one another is determined as a defect candidate 7 (Step S21)”, see also para [0053]; “the sum of the absolute values of values of pixels included in the defect candidate 7 (hereinafter, referred to as "evaluation value") is obtained”) by accumulating, for pixels corresponding to the portion, pixel-value differences represented in the predicted image data (see para [0061]; “the first evaluation part 53 performs a provisional evaluation on whether each of defect candidates in the areas of the differential image”, see also para [0055]; “the sum of the absolute values of differences between the values of the pixels included in the defect candidate 7 and the average value”) determining, by the processing circuit, whether the cumulative difference value exceeds a predetermined threshold (see para [0054]; “where the evaluation value of each the defect candidate 7 is compared with the defect evaluation threshold value of 6 (Step S22)”, see also para [0055]; “comparing the sum of the absolute values of values of pixels included in the defect candidate 7 with a predetermined threshold value”); and in response to determining that the cumulative difference value exceeds the predetermined threshold, detecting, by the processing circuit, the portion as a defect (see para [0054]; “the defect candidate 7 is provisionally evaluated to be a true defect. In the defect candidate 7 consisting of one defect candidate pixel 71 and one quasi-defect candidate pixel 72 which are connected with one another, since the evaluation value is not larger than the defect evaluation threshold value of 6 depending on the value of the defect candidate pixel 71, in this case, the defect candidate 7 is provisionally evaluated to be false detection”). The combination of Bunno et al., Rothmund et al, Namiki, Kimura et al. Majurski et al. and Matsumura a whole does not teach the portion as a defect that is not any of the types of defects and classifying the defect as an unknown defect.
In the same field of endeavor, Hyatt et al teach the portion as a defect that is not any of the types of defects and classifying the defect as an unknown defect (see para [0024]; “defects on imaged items are detected by inputting to a processor trained on images of items of a first type, an image of a defect free item of a second type and an image of an item of the second type of unknown class. The processor is then used to detect a defect on the item of the second type in the image of item of the second type of unknown class”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al., Namiki and Rothmund et al. in view of detecting an anomaly by using the weighted differential dataset of Kimura et al. and exact tile-based segmentation inference for images larger than GPU memory of Majurski et al. and further in view of a method of blending stitched document image portions of Hou et al. and apparatus and method for detecting defect on object of Matsumura and a machine learning based detection system, in which defects can be detected of Hyatt et al. in order offer an inherently flexible detection system (see para [0024]).
Claims 7-10, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Banno et al. and Kimura et al., in view of Majurski et al. and further in view of Hou et al.
Regarding claim 7, Banno et al. teaches a defect detection method see para [0050]; “The computer 32 inspects the item 12 for defects using the image data sent from the camera 31”) comprising steps of: preparing a learned machine learning model by a processing circuit performing learning using, as teaching data, original image data obtained by imaging of an inspection target object and marked image data (see para [0020]; “an identifier is generated by performing supervised or semi-supervised machine learning using labeled image data as teaching data, the labeled image data being generated by the method for generating labeled image data”, see also para [0055]; “the original images used to generate teaching data are not limited to the ones captured by the camera 31 of the inspection device 30 and may be images of the item 12 captured in advance at a different location by a camera that is separate from the inspection device 30”), wherein tiling processing is performed on the original image data and the marked image data to obtain segmented images, learning of a correspondence relationship between the original image data and the marked image data is performed for each tiled segmented image (see para [0020]; “an identifier is generated by performing supervised or semi-supervised machine learning using labeled image data as teaching data, the labeled image data being generated by the method for generating labeled image data. In the determination step, multiple image segments are obtained by segmenting an image of an inspection subject into the number of segments that is the same as the number of segments of the image segments used for generating the identifier”), and, in the tiling processing, segmented images are created such that, after tiling, both edge portions of adjacent segmented images overlap (see para [0014]; “the method for generating labeled image data, it is preferred that in the segmentation step, the image including the other channel be segmented into the multiple image segments in which adjacent image segments are partially overlapped with each other”); acquiring, by the processing circuit, labeled image data of an inspection target object output by the learned machine learning model by inputting image data of the inspection target object to the machine learning model (see para [0028]; “The determination unit determines whether the inspection subject is defective based on an output result from the identifier when the image segments obtained by the image segment acquisition unit is input to the identifier…”), the input image data having been subjected to the tiling processing (see para [0026]; “The segmentation unit divides the image including the other channel into multiple image segments”). However, Banno et al. does not teach in which a defective portion having a contour shape and a defective portion having a region shape that may be present in the inspection target object are labeled with a predetermined color for each type of defect, acquiring, by the processing circuit, predicted image data showing a detection performing reverse tiling processing, by the processing circuit, on the predicted image data, the reverse tiling processing performing combination by ignoring a portion of an outer half of an overlapping edge portion in each piece of the predicted image data before combination, wherein, in the step of performing the reverse tiling processing, hue, saturation, and brightness are adjusted based on a predetermined color for each type of defect between a plurality of pieces of the predicted image data before combination.
In the same field of endeavor, Namiki teaches in which a defective portion having a contour shape and a defective portion having a region shape that may be present in the inspection target object are labeled with a predetermined color for each type of defect (see para [0060]; “a predetermined color may be given so as to surround the defect 20 on the surface of the inspection target 12 by a predetermined color pen…a predetermined shape may be employed as the symbol 22. For example, it is possible to employ various shapes such as the graphic of a rectangle, or a star shape. The defect 20 may be surrounded by the graphic of a rectangle, or a star shape may be drawn on the site of the defect 20”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al. in view of a machine learning device that creates training data to be used in machine learning of Namiki in order to distinguish between OK and NG on new input images of the inspection target (see para [0060]). However, the combination of Bunnno et al. and Namiki as a whole does not teach acquiring, by the processing circuit, predicted image data showing a detection performing reverse tiling processing, by the processing circuit, on the predicted image data, the reverse tiling processing performing combination by ignoring a portion of an outer half of an overlapping edge portion in each piece of the predicted image data before combination.
In the same field of endeavor, Majurski et al. teaches acquiring, by the processing circuit, predicted image data showing a detection performing reverse tiling processing, by the processing circuit, on the predicted image data, the reverse tiling processing performing combination by ignoring a portion of an outer half of an overlapping edge portion in each piece of the predicted image data before combination (see page 1, Abstract; “Our approach is to select a tile size that will fit into GPU memory with a halo border of half the network receptive field. Next, stride across the image by that tile size without the halo. The input tile halos will overlap, while the output tiles join exactly at the seams. Such an approach enables inference to be performed on whole slide microscopy images, such as those generated by a slide scanner”, see also page 5, 2nd para; “The ZoR needs to be cropped out from the prediction so only those pixels with full context are used to build the final result”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al. in view of a machine learning device that creates training data to be used in machine learning of Namiki and exact tile-based segmentation inference for images larger than GPU memory of Majurski et al. in order to estimate the tiling parameters (see Abstract). However, the combination of Bunnno et al., Namiki and Majurski et al. as a whole does not teach wherein, in the step of performing the reverse tiling processing, hue, saturation, and brightness are adjusted based on a predetermined color for each type of defect between a plurality of pieces of the predicted image data before combination.
In the same field of endeavor, Hou et al. teaches wherein, in the step of performing the reverse tiling processing, hue, saturation, and brightness are adjusted based on a predetermined color for each type of defect between a plurality of pieces of the predicted image data before combination (see para [0036]; “the categorization is undertaken by analyzing the brightness and saturation of pixels…. the brightness value (1 component in the HIS color model) and saturation value (S component in the HIS colour model)”, see also para [0044]; “The new HIS value of P0 is P0+DIFF/2, and the new HIS value of its corresponding pixel P1 is P1-DIFF/2”, and para [0057]; “The blended image portions are combined”). Accordingly, it would have been obvious to one ordinary skill in the art of the claimed invention before the effective filling date of the general use of a method for generating labeled image data of Bunno et al. in view of a machine learning device that creates training data to be used in machine learning of Namiki and exact tile-based segmentation inference for images larger than GPU memory of Majurski et al. and further in view of a method of blending stitched document image portions of Hou et al. in order to reduce visually perceivable seams (see para [0036]).
Regarding claim 8, the rejection of claim 7 is incorporated herein.
Bunno et al. in the combination further teach further comprising a step of: classifying a defect in the image data of the inspection target object according to the predetermined color included in the predicted image data (see para [0012]; “a different second pixel value is assigned in accordance with the type of the defect. Further, it is preferred that in the label assignment step, a different defect label be assigned to the image segment including the second pixel value in accordance with the value of the second pixel value”).
Regarding claim 9, the rejection of claim 7 is incorporated herein.
Namiki in the combination further teach further comprising a step of: in a case where the predicted image data includes a portion of a color that is not any of the predetermined colors, classifying the portion of the color as a portion in which a defect that is not any of the types of defects is detected.
Regarding claim 10, the rejection of claim 7 is incorporated herein.
Namiki in the combination further teach wherein the machine learning model includes a convolutional neural network (CNN) or a fully convolutional neural network (FCNN) (see para [0054]; “CNN (Convolutional Neural Network) can also be applied”).
Regarding claim 15, the rejection of claim 7 is incorporated herein.
Namiki in the combination further teach a storage medium storing a computer program causing the processing circuit to execute the defect detection method (see para [0007]; “computer program product storing instructions embodied on a computer-readable medium or programmable circuitry, that, when executed by a processor or the programmable circuitry, cause the processor or the programmable circuitry to perform the method”).
Regarding claim 18, the rejection of claim 7 is incorporated herein.
Namiki in the combination further teach a defect detection system comprising a computer device and a storage device, wherein the computer device performs the defect detection method (see para [0007]; “computer program product storing instructions embodied on a computer-readable medium or programmable circuitry, that, when executed by a processor or the programmable circuitry, cause the processor or the programmable circuitry to perform the method”).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WINTA GEBRESLASSIE whose telephone number is (571)272-3475. The examiner can normally be reached Monday-Friday9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WINTA GEBRESLASSIE/Examiner, Art Unit 2677