DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . The preliminary Amendment filed 04 December 2024 has been entered and considered. Claims 1, 11, and 13 have been amended. Claim 2 has been cancelled. Claims 1 and 3-13 are all the claims pending in the application.
Priority
This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of Application No. JP2022-097546, filed in Japan on 06/16/2022.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/04/2024 was considered by the examiner.
Claim Objections
Claims 3-4 are objected to as being dependent upon cancelled Claim 2. For purposes of compact persecution, Claims 3-4 are being treated as if dependent upon Claim 1. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-6, and 9-13 are rejected under 35 U.S.C. 103 as being unpatentable over Chung et al. (U.S. Patent App. Pub No. 2019/0272627 A1, hereafter referred as Chang) in view of He et al. (NPL: ADASYN: Adaptive synthetic sampling approach for imbalanced learning, hereafter referred as He).
Regarding Claim 1:
Chung teaches a visual inspection device having a processor (Chung: Par. [0001]; the present invention relates to the generation of image data sets that can be used in training systems for defect detection, Par. [0045]; The system 10 has a processor 20), wherein the processor obtains a plurality of visual images of visual of an object to be inspected (Chung: Par. [0047]; Referring to FIG. 2, two original images of the same manufacturing defect is illustrated. The left image is a transparent image of the feature of interest while the right image is a reflective image of the defect. It should be clear that multiple images of the same defect or feature of interest may be used as the process is similar regardless of the type of original image used.), generates a statistical distribution expressing variation in a characteristic of each of the visual images when the plurality of visual images are set as a population (Chung: Par. [0046]; Once the clean image has been created or obtained, and once the feature image has been isolated, the characteristics of the specific feature of interest are then determined. Characteristics of similar features (i.e. similar defects) can then be added to a list of the characteristics, Par. [0055]; Of course, if multiple feature images are available (i.e. multiple original images are being used), this list of characteristics may be lengthy with each feature image having its own list of characteristics. For such an embodiment, all the various characteristics from all of the multiple feature images from the various original images are collated into a single characteristic list. Of course, the single characteristic list would only be compiled if all of the original images are of features that are of the same type, class, configuration, or even orientation as desired by the user.), generates an additional image of the visual on the basis of the variation indicated by the statistical distribution (Chung: Par. [0046]; Based on these characteristics and based on randomly generated characteristics, images of similar features can then be generated.), and generates a learned model by machine learning using learning data including the plurality of visual images and the additional images (Chung: Par. [0046]; Once generated, these new feature images can then be combined to result in new images that can be used in data sets for training AI systems in defect recognition and detection.).
Chung fails to further teach wherein the larger the variation in a distribution region in the statistical distribution is, the more the processor increases the number of additional images for the distribution region.
He, like Chung, is directed to an image generating method. He does teach wherein the larger the variation in a distribution region in the statistical distribution is, the more the processor increases the number of additional images for the distribution region (He: Introduction; more synthetic data is generated for minority class samples that are harder to learn compared to those minority samples that are easier to learn, ADASYN Algorithm; The key idea of ADASYN algorithm is to use a density distribution r^i as a criterion to automatically decide the number of synthetic samples that need to be generated for each minority data example. Physically, r^i is a measurement of the distribution of weights for different minority class examples according to their level of difficulty in learning. The resulting dataset post ADASYN will not only provide a balanced representation of the data distribution (according to the desired balance level defined by the β coefficient), but it will also force the learning algorithm to focus on those difficult to learn examples.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung to the adaptive synthetic sampling approach, as taught by He, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods for improvement to yield predictable results. As taught by He, the proposed modification can not only reduce the learning bias introduced by the original imbalance data distribution, but can also adaptively shift the decision boundary to focus on those difficult to learn samples (He: Introduction).
In regards to Claim 3, Chung as modified by He further teaches the visual inspection device according to claim [1], wherein the characteristic is a deformation amount of the object to be inspected appearing in the visual image (Chung: Par. [0046]; Once the clean image has been created or obtained, and once the feature image has been isolated, the characteristics of the specific feature of interest are then determined. Characteristics of similar features (i.e. similar defects) can then be added to a list of the characteristics.).
In regards to Claim 4, Chung as modified by He further teaches the visual inspection device according to claim [1], wherein the characteristic is a position where a deformation occurs in the object to be inspected appearing in the visual image, and the processor generates the additional image by superimposing a defect on the position in the visual image (Chung: Par. [0064]; The feature (i.e. the defect) is then extracted from the original image (step 140) and its characteristics determined (step 150). Additional characteristics for similar features can then be added to the characteristics list (step 160). Based on the augmented characteristics list (along with possibly some Gaussian noise parameter), numerous feature images are then generated (step 170). These generated feature images are then combined with the clean image to result in images which can be used for training.).
In regards to Claim 5, Chung as modified by He further teaches the visual inspection device according to claim 1, wherein the smaller the number of visual images in a distribution region in the statistical distribution is, the more the processor increases the number of the additional images for the distribution region (He: Introduction; more synthetic data is generated for minority class samples that are harder to learn compared to those minority samples that are easier to learn, ADASYN Algorithm; The key idea of ADASYN algorithm is to use a density distribution r^i as a criterion to automatically decide the number of synthetic samples that need to be generated for each minority data example. Physically, r^i is a measurement of the distribution of weights for different minority class examples according to their level of difficulty in learning. The resulting dataset post ADASYN will not only provide a balanced representation of the data distribution (according to the desired balance level defined by the β coefficient), but it will also force the learning algorithm to focus on those difficult to learn examples.).
In regards to Claim 6, Chung as modified by He further teaches the visual inspection device according to claim 5, wherein the characteristic is any of luminance, contrast, and noise intensity of the visual image (Chung: Par. [0055]; From the feature image, the characteristics of the features (i.e. the defects in this example) can then be determined (see FIG. 5). Accordingly, the color, shape, type of edges of the feature (i.e. edge style), direction, the size of the feature (relative to the pixel size in this example), as well as other characteristics, can be found. A suitable process for extracting specific characteristics about the feature can be formulated by a person of skill in the art. It should be clear that such a process may include specifically detailing the characteristics being extracted or determined. Thus, the characteristics may be specific to the type of original image being used (e.g. if it is a reflective image, the color may be “thin” such that it looks like a black and white image while an original transparent image may have a full spectrum of available colors) as well as the scale of the original image (e.g. if the original image is large, then the scale of the feature may be based on a scale different from a pixel scale).).
In regards to Claim 9, Chung as modified by He further teaches the visual inspection device according to claim 1, wherein each of the plurality of visual images is an image of the visual of the object to be inspected which is normal (Chung: Par. [0051]; With the background section extracted, a gridded image is created and the extracted section is then replicated into each of the various grids in the gridded image. In other words, the section is tiled across the gridded image to result in a clean image, i.e. an image that does not include the feature or defect but which includes the background of the original image. In FIG. 3, the clean images from the transparent and reflective images are on the right side of the feature while the original images are on the left side of the Figure.).
In regards to Claim 10, Chung as modified by He further teaches the visual inspection device according to claim 1, wherein each of the plurality of visual images is an image of the visual of the object to be inspected which is abnormal (Chung: Par. [0047]; Referring to FIG. 2, two original images of the same manufacturing defect is illustrated. The left image is a transparent image of the feature of interest while the right image is a reflective image of the defect. It should be clear that multiple images of the same defect or feature of interest may be used as the process is similar regardless of the type of original image used.).
In regards to Claim 11, Chung as modified by He further teaches the visual inspection device according to claim 9, wherein the processor processes an image of the visual of the object to be inspected which is normal, to generate the additional image indicating a normality (Chung: Par. [0051]; With the background section extracted, a gridded image is created and the extracted section is then replicated into each of the various grids in the gridded image. In other words, the section is tiled across the gridded image to result in a clean image, i.e. an image that does not include the feature or defect but which includes the background of the original image. In FIG. 3, the clean images from the transparent and reflective images are on the right side of the feature while the original images are on the left side of the Figure.).
In regards to Claim 12, Chung as modified by He further teaches the visual inspection device according to claim 9, wherein the processor processes an image of the visual of the object to be inspected which is normal or abnormal, to generate the additional image indicating an abnormality (Chung: Par. [0054]; The next step is to isolate the feature or defect from the original image. As can be seen from FIG. 4, this can be done with the help of the clean image generated or obtained previously. A simple image operation of subtracting the clean image from the original image results in a feature image that consists only of the feature or defect from the original image. Other steps to clean up or render more clearly the feature image (e.g. denoising the resulting feature image) may be carried out as well. Of course, other methods for extracting or isolating the feature or defect can also be used.).
Regarding Claim 13:
Chung as modified by He further teaches an image generating method that makes a computer execute (Chung: Par. [0001]; the present invention relates to the generation of image data sets that can be used in training systems for defect detection, Par. [0045]; The system 10 has a processor 20): a step of obtaining a plurality of visual images of visual of an object to be inspected (Chung: Par. [0047]; Referring to FIG. 2, two original images of the same manufacturing defect is illustrated. The left image is a transparent image of the feature of interest while the right image is a reflective image of the defect. It should be clear that multiple images of the same defect or feature of interest may be used as the process is similar regardless of the type of original image used.); a step of generating a statistical distribution expressing variation of a characteristic of each of the visual images when the plurality of visual images are set as a population (Chung: Par. [0046]; Once the clean image has been created or obtained, and once the feature image has been isolated, the characteristics of the specific feature of interest are then determined. Characteristics of similar features (i.e. similar defects) can then be added to a list of the characteristics, Par. [0055]; Of course, if multiple feature images are available (i.e. multiple original images are being used), this list of characteristics may be lengthy with each feature image having its own list of characteristics. For such an embodiment, all the various characteristics from all of the multiple feature images from the various original images are collated into a single characteristic list. Of course, the single characteristic list would only be compiled if all of the original images are of features that are of the same type, class, configuration, or even orientation as desired by the user.); a step of generating an additional image of the visual on the basis of the variation indicated by the statistical distribution (Chung: Par. [0046]; Based on these characteristics and based on randomly generated characteristics, images of similar features can then be generated.); and a step of generating a learned model by machine learning using learning data including the plurality of visual images and the additional images (Chung: Par. [0046]; Once generated, these new feature images can then be combined to result in new images that can be used in data sets for training AI systems in defect recognition and detection.), wherein the larger the variation in a distribution region in the statistical distribution is, the more the processor increases the number of additional images for the distribution region (He: Introduction; more synthetic data is generated for minority class samples that are harder to learn compared to those minority samples that are easier to learn, ADASYN Algorithm; The key idea of ADASYN algorithm is to use a density distribution r^i as a criterion to automatically decide the number of synthetic samples that need to be generated for each minority data example. Physically, r^i is a measurement of the distribution of weights for different minority class examples according to their level of difficulty in learning. The resulting dataset post ADASYN will not only provide a balanced representation of the data distribution (according to the desired balance level defined by the β coefficient), but it will also force the learning algorithm to focus on those difficult to learn examples.).
Claims 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Chung et al. (U.S. Patent App. Pub No. 2019/0272627 A1, hereafter referred as Chang) in view of He et al. (NPL: ADASYN: Adaptive synthetic sampling approach for imbalanced learning, hereafter referred as He) and Beggel et al. (NPL: Robust Anomaly Detection in Images Using Adversarial Autoencoders, hereafter referred as Beggel).
In regards to Claim 7, Chung as modified by He fails to further teach the visual inspection device according to claim 1, wherein the processor further inspects whether there is an abnormality in the object to be inspected which is appearing in an inspection image by using the learned model.
Beggel, like Chung, is directed to visual inspection. Beggel does teach wherein the processor further inspects whether there is an abnormality in the object to be inspected which is appearing in an inspection image by using the learned model (Beggel: Introduction; we define an iteration refinement method for training sample rejection. Potential anomalies in the training set are identified in the lower dimensional latent space by a variation of 1-class SVM [18], and by rejecting the least normal observations we can increase robustness to contaminated data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chung to utilize the adversarial autoencoders, as taught by Beggel, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods for improvement to yield predictable results. As taught by Beggel, the proposed modification allows for the combination of likelihood and reconstruction error to yield an improved anomaly score and therefore better detection performance (Beggel: Introduction).
In regards to Claim 8, Chung as modified by He and Beggel further teaches the visual inspection device according to claim 7, wherein the learned model is an autoencoder which learns the learning data as correct data (Beggel: Training with Anomalies; Instead, the AE is trained to minimize reconstruction errors on the entire training set, which will only directly optimize the first criterion if all training images are normal. During training, the objective rewards exact reconstructions of all training images, including anomalies.), and the processor enters the inspection image to the autoencoder and determines whether a foreign matter is included in a differential image as the difference between a reconstructed image output from the autoencoder and the inspection image, thereby inspecting whether or not there is an abnormality in the object to be inspected (Beggel: Robust Anomaly Detection; Separation between anomalies and normal instances in latent space is particularly useful if a rough estimate of the training anomaly rate alpha is known. In this case standard outlier detection methods such as 1-class SVM [18] can be employed on the latent representations, searching for a boundary that contains a fraction of 1 – alpha of the whole dataset. Once potential anomalies are identified, they can be excluded for further training, or their contribution to the total loss might be reweighted. Such procedure approximates the case of a clean training set, where the combination of reconstruction error and latent density yields reliable results.).
Pertinent Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Okano et al. (U.S. Patent App. Pub No. 2023/0351730 A1) teaches a device that groups similar defect images together.
Wu et al. (U.S. Patent App. Pub No. 2021/0073972 A1) teaches an automated defect-detection platform for inspecting items in a physical environment, such as semiconductor wafers.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RENAE BITOR whose telephone number is (703)756-5563. The examiner can normally be reached Monday to Friday: 8:00 - 5:30 but off the 1st Friday of the biweek.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, GREG MORSE can be reached on (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RENAE A BITOR/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights