Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on April 16, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant's arguments filed 3/26/2026 have been fully considered but they are not persuasive.
Applicant argues that the prior art of record, Nagato et. al. fails to teach the machine learning model as disclosed by the present application in independent claim 1. Nagato et. al. provides an approach based on genetic programming; however, examiner argues that this does not change the solution or result of the machine learning model. Genetic algorithms are optimization techniques inspired by the principles of natural selection and genetics, similar to how neural networks are computational models inspired by the human brain. Solutions are evolved through processes analogous to biological evolution, and genetics, and thus this new application serves as another way to utilize these features. The “re-training” feature is effectively shown by optimizing parameters through a series of stages. The fitness score is used to decide which programs “survive” to the next generation, which essentially is construed as the specific data points that are included, which is feedback into the same genetic algorithm model for further refinement.
Thus, the prior art of record still is effective in rejecting all claims.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 8, 9, and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Nagato et. al. (US Patent 2020/0202225 A1).
Regarding claim 1, Nagato discloses an image processing device comprising a processor configured to input an image to a machine learning model and execute classification of classifying the image into a plurality of classes (Abstract, Figure 6, 7, 9, image inspection apparatus 10, [0031] here the function of the image inspection apparatus 10 is outlined with the ability to calculate local feature quantities from learning data (input image) used for the image processing program selected as a determination target, and learns the discriminator by unsupervised learning. The key feature shown by the prior art is the utilization of unsupervised learning, a type of machine learning model that does not rely on prior labeling, through the discriminator which enables the separation of defective regions and non-defective regions. This is then carried out by the learning unit 22),
PNG
media_image1.png
352
556
media_image1.png
Greyscale
wherein the processor: (1) trains the machine learning model with a plurality of training images (Figure 1, 4, 6, 7, [0026]-[0027], [0041]-[0043] teaching unit 21. Note that the teaching unit 21 associates the input image as a whole from the extraction source with the non-defective region data and trains the images based on this data set);
PNG
media_image2.png
244
556
media_image2.png
Greyscale
(2) inputs a plurality of validation images to the machine learning model trained with the plurality of training images, and executes classification of classifying the plurality of validation images into the plurality of classes ([0028]);
(3) obtains a degree of separation between the plurality of classes by the classification of the plurality of validation images and evaluates accuracy of the classification of the plurality of validation images based on the obtained degree of separation between the plurality of classes ([0031]);
and (4) evaluates whether re-training of the machine learning model is necessary based on an evaluation result of the accuracy of classification of the plurality of validation images ([0032]),
extracts a validation image whose classification result has a relatively high possibility to be erroneous from among the plurality of validation images to automatically re-train the machine learning model if it is evaluated that the re-training of the machine learning model is necessary (learning unit 22, [0044], [0069]),
and completes the training of the machine learning model if it is evaluated that the re-training of the machine learning model is unnecessary (Figure 6, [0045]-[0046], [0070]).
Regarding claim 2, Nagato et. al. teaches the image processing device according to claim 1, wherein the processor automatically repeats the processing of (2) to (4) until it is evaluated that the re-training of the machine learning model is unnecessary (Figure 1, processor with automatic generation by genetic programming).
Regarding claim 8, Nagato et. al. teaches the image processing device according to claim 1, wherein if it is evaluated that the re-training of the machine learning model is necessary, the processor automatically calculates the number of validation images used for re-training based on an evaluation result of the accuracy of the classification ([0056]),
and extracts the validation images whose classification result has a relatively high possibility to be erroneous by the calculated number from among the plurality of validation images to automatically re-train the machine learning model with the validation images of the corresponding number (learning unit 22, [0044], [0069]).
Regarding claim 9, Nagato et. al. teaches the image processing device according to claim 8, wherein if it is evaluated that the re-training of the machine learning model is necessary, the processor extracts the validation images by the calculated number from among the plurality of validation images in relatively descending order of possibility of the classification result being erroneous to automatically re-train the machine learning model with the validation images of the corresponding number ([0061]).
Regarding claim 15, Nagato et. al. recites an image processing method for inputting an image to a machine learning model and executing classification of classifying the image into a plurality of classes, the image processing method comprising (Figure 1, 7):
(1) a step of training the machine learning model with a plurality of training images (Figure 1, 6, 7, [0026]-[0027], teaching unit 21);
(2) a step of inputting a plurality of validation images to the machine learning model trained with the plurality of training images and executing classification of classifying the plurality of validation images into the plurality of classes ([0028]);
(3) a step of obtaining a degree of separation between the plurality of classes by the classification of the plurality of validation images and evaluating accuracy of the classification of the plurality of validation images based on the obtained degree of separation between the plurality of classes ([0028]);
and (4) a step of evaluating whether re-training of the machine learning model is necessary based on an evaluation result of the accuracy of classification of the plurality of validation images ([0032]),
extracting a validation image whose classification result has a relatively high possibility to be erroneous from among the plurality of validation images to automatically re-train the machine learning model if it is evaluated that the re-training of the machine learning model is necessary (learning unit 22, [0044], [0069]),
and completing the training of the machine learning model if it is evaluated that the re-training of the machine learning model is unnecessary (Figure 6, [0045]-[0046], [0070]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3-7, 10-13 are rejected under 35 U.S.C. 103 as being unpatentable over Nagato et. al. in view of Saha et. al. (US Patent 2022/0318602 A1).
Regarding claim 3 and claim 5, Nagato et. al. teaches the image processing device according to claim 2, wherein the plurality of classes includes a first class and a second class (Nagato et. al.: [0041], [0045]—“as illustrated in Fig.6, the learning unit 22 acquires, as learning data, an input image including a defective portion and non-defective region data corresponding to the input image.” Here, the first class is construed as non-defective product image and second class is construed as defective product image.
the processor calculates a first evaluation value indicating a degree of belonging to the first class for each of the plurality of validation images in the classification of classifying the plurality of validation images into the first class and second class (Nagato et. al. [0055]), and if it is evaluated that the re-training of the machine learning model is necessary,
the processor extracts the second-class image having a relatively high first evaluation value or the first-class image having a relatively low first evaluation value to automatically re-train the machine learning model (Nagato et. al. [0056]),
obtains a degree of separation of distribution between the first-class image and the second-class image on a first evaluation value axis, and repeats the processing of (2) to (4) until it is evaluated that the re-training of the machine learning model is unnecessary based on the degree of separation (Nagato et. al. claim 2).
PNG
media_image3.png
146
431
media_image3.png
Greyscale
However, Nagato et. al. fails to teach the first-class image is given a label indicating the first class and a second-class image is given a label indicating the second class.
Saha et. al. teaches explicitly that the first-class image is given a label indicating the first class and a second-class image is given a label indicating the second class (Saha et. al. [0027], [0035]).
PNG
media_image4.png
294
432
media_image4.png
Greyscale
The image labeling introduced by Saha et. al. is important to the claimed invention so that there is no confusion in the process of classifying a plurality of images during inspection, and is a feature of deep neural network (DNN) which is a type of machine learning model that excels in image labeling tasks. Thus, it would have been obvious to a person having ordinary skills in the art (PHOSITA) before the effective filing date of the claimed invention to have combined the teachings of Nagato et. al. with the teachings of Saha et. al. to incorporate the deep neural network ability to label images of Saha et. al. with the image processing device of Nagato et. al. that discriminates between defective and non-defective classes.
Regarding claim 4, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 3. Nagato et. al. further teaches wherein the processor repeats the processing of (2) to (4) until the degree of separation of distribution between the first-class image and the second-class image on the first evaluation value axis becomes equal or greater than a predetermined value (Nagato et. al. [0056], [0062]).
Regarding claim 6, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 5. Nagato et. al. further teaches wherein the processor automatically determines which of the non-defective product image having a relatively high evaluation value and the defective product image having a relatively low evaluation value is to be extracted as an image for re-training the machine learning model based on a result of comparison between the evaluation value of the non-defective product image and the evaluation value of the defective product image among the plurality of validation images (Nagato et. al. [0056]).
Regarding claim 7, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 3. Nagato et. al. further teaches wherein the processor further calculates a second evaluation value indicating a degree of belonging to the second class for each of the plurality of validation images in the classification (Nagato et. al. evaluation unit 27, [0054], fitness calculation process, Figure. 8),
and if it is evaluated that the re-training of the machine learning model is necessary (Nagato et. al. [0069]-[0070]),
the processor extracts the first-class image having a relatively high second evaluation value or the second-class image having a relatively low second evaluation value to automatically re-train the machine learning model (Nagato et. al. [0069]-[0070]),
obtains a degree of separation between distributions of the first-class image and the second-class image on a second evaluation value axis, and repeats the processing of (2) to (4) until it is evaluated that the re-training of the machine learning model is unnecessary based on the degree of separation (Nagato et. al. claim 2).
Regarding claim 10, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 3. Nagato et. al. further teaches wherein the processor generates a display screen including a separation graph indicating the degree of separation of the distribution between the first-class image and the second-class image on the first evaluation value axis and causes a display unit to display the display screen after executing the classification (Nagato et. al. Figure 6),
and updates the separation graph to the separation graph obtained by the machine learning model after the re-training (Nagato et. al. Figure 7),
generates a display screen displaying a latest separation graph obtained by the update in a manner comparable to a past separation graph, and causes the display unit to display the display screen every time the re-training of the machine learning model is repeated (Nagato et. al. Figure 8).
Regarding claim 11, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 10. Saha et. al. teaches wherein the processor is configured to generate a display screen displaying, in a comparable manner, a latest separation graph obtained by repeatedly re-training the machine learning model and a past separation graph obtained in a training stage before a training stage in which the latest separation graph is obtained, cause the display unit to display the display screen, and receive selection of a past machine learning model corresponding to the past separation graph from a user as a machine learning model to be used in operation (Saha et. al. [0054]-display screen 210A). However, Saha et. al. fails to teach and when selection of the past machine learning model is received, the processor identifies all training images for training the selected past machine learning model, and trains a machine learning model in an initial state before starting training with all the identified training images, thereby reproducing the selected past machine learning model. Nagato et. al. teaches and when selection of the past machine learning model is received, the processor identifies all training images for training the selected past machine learning model, and trains a machine learning model in an initial state before starting training with all the identified training images, thereby reproducing the selected past machine learning model (Nagato et. al. Figure 6).
PNG
media_image5.png
598
858
media_image5.png
Greyscale
It is important for the results of the classification between defective product image and non-defective product image to be displayed on a user-friendly screen so that products can be quickly identified. Thus, it would have been obvious to a PHOSITA prior to the effective filing date of the claimed invention to have incorporated the teachings of Saha et. al. and the teachings of Nagato et. al. to include a display screen of Saha et. al. capable of showing the calculated results of graph from the machine learning process of Nagato et. al.
Regarding claim 12, Nagato et. al. teaches the image processing device according to claim 1, the processor inputs a new validation image different from an existing validation image to the machine learning model and executes a process of classifying the new validation image into the plurality of classes (Nagato et. al. Figure 2, 3),
the processor evaluates accuracy of the classification based on a degree of separation between the plurality of classes obtained by the classification for the new validation image,
and if it is evaluated that the re-training of the machine learning model is necessary based on an evaluation result of the accuracy of the classification,
the processor extracts a validation image whose classification result has a high possibility to be erroneous from among the new validation image to automatically re-train the machine learning model, and on the other hand, if it is evaluated that the re-training of the machine learning model is unnecessary, the processor ends the training of the machine learning model (Nagato et. al. Claim 11).
PNG
media_image6.png
288
528
media_image6.png
Greyscale
However, Nagato et. al. fails to teach wherein the processor is configured to restart the retraining of the machine learning model in response to a user instruction even after completing the training.
Saha et. al. teaches wherein the processor is configured to restart the retraining of the machine learning model in response to a user instruction even after completing the training (Saha et. al. [0053]).
PNG
media_image7.png
277
414
media_image7.png
Greyscale
It is critical to include the feature of allowing user input to the image processing device so that instructions can be initialized by the user based on the results of the trained DNN. Thus, it would have been obvious to a PHOSITA prior to the effective filing date of the claimed invention to include the user provided instructions of Saha et. al. with the teachings of the Nagato et. al.
Regarding claim 13, Nagato et. al. teaches the image processing device according to claim 1. However, Nagato et. al. fails to teach wherein the processor is configured to select: a process of holding in advance labels related to the classification of existing training images and validation images used for the training of the machine learning model in operation and adding a training image by adding a new image acquired after starting the operation to the existing training images;
and a process of initializing the labels related to the classification of the existing training images and validation images used for the training of the machine learning model in operation and training the machine learning model with all images including a new image acquired after starting the operation.
Saha et. al. teaches wherein the processor is configured to select: a process of holding in advance labels related to the classification of existing training images and validation images used for the training of the machine learning model in operation and adding a training image by adding a new image acquired after starting the operation to the existing training images;
and a process of initializing the labels related to the classification of the existing training images and validation images used for the training of the machine learning model in operation and training the machine learning model with all images including a new image acquired after starting the operation (Saha et. al. [0035]).
PNG
media_image8.png
538
406
media_image8.png
Greyscale
It is critical to include the advanced labeling capabilities of the DNN model as taught by Saha et. al. to minimize potential confusion among the plurality of image datasets for product inspection. Thus, it would be obvious to a PHOSITA prior to the effective filing date of the claimed invention to combine the teachings of Nagato et. al. to include the advanced labeling techniques of Saha et. al.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Nagato et. al. in view of Bian et. al. (US Patent 2021/0133511 A1).
Regarding claim 14, Nagato et. al. teaches the image processing device according to claim 1. However, Nagato et. al. fails to teach wherein if it is evaluated that the re-training of the machine learning model is unnecessary, after completing the training of the machine learning model, the processor inputs a plurality of evaluation images different from the plurality of training images and the plurality of validation images to the machine learning model, executes classification of classifying the plurality of evaluation images into the plurality of classes, and evaluates accuracy of the classification.
Bian et. al. teaches re-training of “tail-classes” e.g., class of images that did not meet a threshold test during inspection in order to expand the sample images for more accurate defect detection. Thus, Bian et. al. teaches wherein if it is evaluated that the re-training of the machine learning model is unnecessary, after completing the training of the machine learning model, the processor inputs a plurality of evaluation images different from the plurality of training images and the plurality of validation images to the machine learning model, the processor inputs a plurality of evaluation images different from the plurality of training images and the plurality of validation images to the machine learning model, , executes classification of classifying the plurality of evaluation images into the plurality of classes, and evaluates accuracy of the classification (classification module, Figure 3, [0017], [0018]). Thus, it would be obvious to a PHOSITA prior to the effective filing date of the claimed invention to combine the teachings of Nagato et. al. to include the techniques of Bian et. al. for expanding inspection images in order to achieve better accuracy during inspection.
PNG
media_image9.png
527
416
media_image9.png
Greyscale
Response to Amendment
Examiner acknowledges the amendment to the specification made such that the objections to the disclosure are now overcome.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
/JESSICA YIFANG LIN/Examiner, Art Unit 2668 April 3, 2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668