Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on April 16, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/26/2026 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-13, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nagato et. al. (US Patent Application Publication 2020/0202225 A1) in view of Saha et. al. (US Patent 2022/0318602 A1).
Regarding claim 1, Nagato discloses an image processing device comprising a processor configured to input an image to a machine learning model and execute classification of classifying the image into a plurality of classes (Abstract, Figure 6, 7, 9, image inspection apparatus 10, [0031] here the function of the image inspection apparatus 10 is outlined with the ability to calculate local feature quantities from learning data (input image) used for the image processing program selected as a determination target, and learns the discriminator by unsupervised learning. The key feature shown by the prior art is the utilization of unsupervised learning, a type of machine learning model that does not rely on prior labeling, through the discriminator which enables the separation of defective regions and non-defective regions. This is then carried out by the learning unit 22),
PNG
media_image1.png
352
556
media_image1.png
Greyscale
wherein the processor: (1) trains the machine learning model with a plurality of training images (Figure 1, 4, 6, 7, [0026]-[0027], [0041]-[0043] teaching unit 21. Note that the teaching unit 21 associates the input image as a whole from the extraction source with the non-defective region data and trains the images based on this data set);
PNG
media_image2.png
244
556
media_image2.png
Greyscale
(2) inputs a plurality of validation images to the machine learning model trained with the plurality of training images, and executes classification of classifying the plurality of validation images into the plurality of classes ([0028]);
(3) obtains a degree of separation between the plurality of classes by the classification of the plurality of validation images and evaluates accuracy of the classification of the plurality of validation images based on the obtained degree of separation between the plurality of classes ([0031]);
and (4) evaluates whether re-training of the machine learning model is necessary based on an evaluation result of the accuracy of classification of the plurality of validation images ([0032]),
extracts a validation image whose classification result has a relatively high possibility to be erroneous from among the plurality of validation images to automatically re-train the machine learning model if it is evaluated that the re-training of the machine learning model is necessary (learning unit 22, [0044], [0069]),
and completes the training of the machine learning model if it is evaluated that the re-training of the machine learning model is unnecessary (Figure 6, [0045]-[0046], [0070]).
However, Nagato fails to disclose given a label indicating which of the plurality of classes the validation image belongs to.
Saha et. al. teaches explicitly that the first-class image is given a label indicating the first class and a second-class image is given a label indicating the second class (Saha et. al. [0027], [0035]).
PNG
media_image3.png
294
432
media_image3.png
Greyscale
The image labeling introduced by Saha et. al. is important to the claimed invention so that there is no confusion in the process of classifying a plurality of images during inspection, and is a feature of deep neural network (DNN) which is a type of machine learning model that excels in image labeling tasks. Thus, it would have been obvious to a person having ordinary skills in the art (PHOSITA) before the effective filing date of the claimed invention to have combined the teachings of Nagato et. al. with the teachings of Saha et. al. to incorporate the deep neural network ability to label images of Saha et. al. with the image processing device of Nagato et. al. that discriminates between defective and non-defective classes.
Regarding claim 2, Nagato et. al. further teaches the image processing device according to claim 1, wherein the processor automatically repeats the processing of (2) to (4) until it is evaluated that the re-training of the machine learning model is unnecessary (Figure 1, processor with automatic generation by genetic programming).
Regarding claim 3 and claim 5, Nagato et. al. teaches the image processing device according to claim 2, wherein the plurality of classes includes a first class and a second class (Nagato et. al.: [0041], [0045]—“as illustrated in Fig.6, the learning unit 22 acquires, as learning data, an input image including a defective portion and non-defective region data corresponding to the input image.” Here, the first class is construed as non-defective product image and second class is construed as defective product image.
the processor calculates a first evaluation value indicating a degree of belonging to the first class for each of the plurality of validation images in the classification of classifying the plurality of validation images into the first class and second class (Nagato et. al. [0055]), and if it is evaluated that the re-training of the machine learning model is necessary,
the processor extracts the second-class image having a relatively high first evaluation value or the first-class image having a relatively low first evaluation value to automatically re-train the machine learning model (Nagato et. al. [0056]),
obtains a degree of separation of distribution between the first-class image and the second-class image on a first evaluation value axis, and repeats the processing of (2) to (4) until it is evaluated that the re-training of the machine learning model is unnecessary based on the degree of separation (Nagato et. al. claim 2).
PNG
media_image4.png
146
431
media_image4.png
Greyscale
However, Nagato et. al. fails to teach the first-class image whose label indicates the first class and a second-class image whose label indicates the second class.
Saha et. al. teaches explicitly that the first-class image whose label indicates the first class and a second-class image whose label indicates the second class (Saha et. al. [0027], [0035]).
PNG
media_image3.png
294
432
media_image3.png
Greyscale
The image labeling introduced by Saha et. al. is important to the claimed invention so that there is no confusion in the process of classifying a plurality of images during inspection, and is a feature of deep neural network (DNN) which is a type of machine learning model that excels in image labeling tasks. Thus, it would have been obvious to a person having ordinary skills in the art (PHOSITA) before the effective filing date of the claimed invention to have combined the teachings of Nagato et. al. with the teachings of Saha et. al. to incorporate the deep neural network ability to label images of Saha et. al. with the image processing device of Nagato et. al. that discriminates between defective and non-defective classes.
Regarding claim 4, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 3. Nagato et. al. further teaches wherein the processor repeats the processing of (2) to (4) until the degree of separation of distribution between the first-class image and the second-class image on the first evaluation value axis becomes equal or greater than a predetermined value (Nagato et. al. [0056], [0062]).
Regarding claim 6, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 5. Nagato et. al. further teaches wherein the processor automatically determines which of the non-defective product image having a relatively high evaluation value and the defective product image having a relatively low evaluation value is to be extracted as an image for re-training the machine learning model based on a result of comparison between the evaluation value of the non-defective product image and the evaluation value of the defective product image among the plurality of validation images (Nagato et. al. [0056]).
Regarding claim 7, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 3. Nagato et. al. further teaches wherein the processor further calculates a second evaluation value indicating a degree of belonging to the second class for each of the plurality of validation images in the classification (Nagato et. al. evaluation unit 27, [0054], fitness calculation process, Figure. 8),
and if it is evaluated that the re-training of the machine learning model is necessary (Nagato et. al. [0069]-[0070]),
the processor extracts the first-class image having a relatively high second evaluation value or the second-class image having a relatively low second evaluation value to automatically re-train the machine learning model (Nagato et. al. [0069]-[0070]),
obtains a degree of separation between distributions of the first-class image and the second-class image on a second evaluation value axis, and repeats the processing of (2) to (4) until it is evaluated that the re-training of the machine learning model is unnecessary based on the degree of separation (Nagato et. al. claim 2).
Regarding claim 8, Nagato et. al. further teaches the image processing device according to claim 1, wherein if it is evaluated that the re-training of the machine learning model is necessary, the processor automatically calculates the number of validation images used for re-training based on an evaluation result of the accuracy of the classification ([0056]),
and extracts the validation images whose classification result has a relatively high possibility to be erroneous by the calculated number from among the plurality of validation images to automatically re-train the machine learning model with the validation images of the corresponding number (learning unit 22, [0044], [0069]).
Regarding claim 9, Nagato et. al. further teaches the image processing device according to claim 8, wherein if it is evaluated that the re-training of the machine learning model is necessary, the processor extracts the validation images by the calculated number from among the plurality of validation images in relatively descending order of possibility of the classification result being erroneous to automatically re-train the machine learning model with the validation images of the corresponding number ([0061]).
Regarding claim 10, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 3. Nagato et. al. further teaches wherein the processor generates a display screen including a separation graph indicating the degree of separation of the distribution between the first-class image and the second-class image on the first evaluation value axis and causes a display unit to display the display screen after executing the classification (Nagato et. al. Figure 6),
and updates the separation graph to the separation graph obtained by the machine learning model after the re-training (Nagato et. al. Figure 7),
generates a display screen displaying a latest separation graph obtained by the update in a manner comparable to a past separation graph, and causes the display unit to display the display screen every time the re-training of the machine learning model is repeated (Nagato et. al. Figure 8).
Regarding claim 11, the combination of Nagato et. al. and Saha et.al. teach the image processing device according to claim 10. Saha et. al. teaches wherein the processor is configured to generate a display screen displaying, in a comparable manner, a latest separation graph obtained by repeatedly re-training the machine learning model and a past separation graph obtained in a training stage before a training stage in which the latest separation graph is obtained, cause the display unit to display the display screen, and receive selection of a past machine learning model corresponding to the past separation graph from a user as a machine learning model to be used in operation (Saha et. al. [0054]-display screen 210A). However, Saha et. al. fails to teach and when selection of the past machine learning model is received, the processor identifies all training images for training the selected past machine learning model, and trains a machine learning model in an initial state before starting training with all the identified training images, thereby reproducing the selected past machine learning model. Nagato et. al. teaches and when selection of the past machine learning model is received, the processor identifies all training images for training the selected past machine learning model, and trains a machine learning model in an initial state before starting training with all the identified training images, thereby reproducing the selected past machine learning model (Nagato et. al. Figure 6).
PNG
media_image5.png
598
858
media_image5.png
Greyscale
It is important for the results of the classification between defective product image and non-defective product image to be displayed on a user-friendly screen so that products can be quickly identified. Thus, it would have been obvious to a PHOSITA prior to the effective filing date of the claimed invention to have incorporated the teachings of Saha et. al. and the teachings of Nagato et. al. to include a display screen of Saha et. al. capable of showing the calculated results of graph from the machine learning process of Nagato et. al.
Regarding claim 12, Nagato et. al. teaches the image processing device according to claim 1, the processor inputs a new validation image different from an existing validation image to the machine learning model and executes a process of classifying the new validation image into the plurality of classes (Nagato et. al. Figure 2, 3),
the processor evaluates accuracy of the classification based on a degree of separation between the plurality of classes obtained by the classification for the new validation image,
and if it is evaluated that the re-training of the machine learning model is necessary based on an evaluation result of the accuracy of the classification,
the processor extracts a validation image whose classification result has a high possibility to be erroneous from among the new validation image to automatically re-train the machine learning model, and on the other hand, if it is evaluated that the re-training of the machine learning model is unnecessary, the processor ends the training of the machine learning model (Nagato et. al. Claim 11).
PNG
media_image6.png
288
528
media_image6.png
Greyscale
However, Nagato et. al. fails to teach wherein the processor is configured to restart the retraining of the machine learning model in response to a user instruction even after completing the training.
Saha et. al. teaches wherein the processor is configured to restart the retraining of the machine learning model in response to a user instruction even after completing the training (Saha et. al. [0053]).
PNG
media_image7.png
277
414
media_image7.png
Greyscale
It is critical to include the feature of allowing user input to the image processing device so that instructions can be initialized by the user based on the results of the trained DNN. Thus, it would have been obvious to a PHOSITA prior to the effective filing date of the claimed invention to include the user provided instructions of Saha et. al. with the teachings of the Nagato et. al.
Regarding claim 13, Nagato et. al. teaches the image processing device according to claim 1. However, Nagato et. al. fails to teach wherein the processor is configured to select: a process of holding in advance labels related to the classification of existing training images and validation images used for the training of the machine learning model in operation and adding a training image by adding a new image acquired after starting the operation to the existing training images; and a process of initializing the labels related to the classification of the existing training images and validation images used for the training of the machine learning model in operation and training the machine learning model with all images including a new image acquired after starting the operation.
Saha et. al. teaches wherein the processor is configured to select: a process of holding in advance labels related to the classification of existing training images and validation images used for the training of the machine learning model in operation and adding a training image by adding a new image acquired after starting the operation to the existing training images;
and a process of initializing the labels related to the classification of the existing training images and validation images used for the training of the machine learning model in operation and training the machine learning model with all images including a new image acquired after starting the operation (Saha et. al. [0035]).
PNG
media_image8.png
538
406
media_image8.png
Greyscale
It is critical to include the advanced labeling capabilities of the DNN model as taught by Saha et. al. to minimize potential confusion among the plurality of image datasets for product inspection. Thus, it would be obvious to a PHOSITA prior to the effective filing date of the claimed invention to combine the teachings of Nagato et. al. to include the advanced labeling techniques of Saha et. al.
Regarding claim 15, Nagato et. al. further recites an image processing method for inputting an image to a machine learning model and executing classification of classifying the image into a plurality of classes, the image processing method comprising (Figure 1, 7):
(1) a step of training the machine learning model with a plurality of training images (Figure 1, 6, 7, [0026]-[0027], teaching unit 21);
(2) a step of inputting a plurality of validation images to the machine learning model trained with the plurality of training images and executing classification of classifying the plurality of validation images into the plurality of classes ([0028]);
(3) a step of obtaining a degree of separation between the plurality of classes by the classification of the plurality of validation images and evaluating accuracy of the classification of the plurality of validation images based on the obtained degree of separation between the plurality of classes ([0028]);
and (4) a step of evaluating whether re-training of the machine learning model is necessary based on an evaluation result of the accuracy of classification of the plurality of validation images ([0032]),
extracting a validation image whose classification result has a relatively high possibility to be erroneous from among the plurality of validation images to automatically re-train the machine learning model if it is evaluated that the re-training of the machine learning model is necessary (learning unit 22, [0044], [0069]),
and completing the training of the machine learning model if it is evaluated that the re-training of the machine learning model is unnecessary (Figure 6, [0045]-[0046], [0070]).
However, Nagato fails to disclose given a label indicating which of the plurality of classes the validation image belongs to.
Saha et. al. teaches explicitly that the first-class image is given a label indicating the first class and a second-class image is given a label indicating the second class (Saha et. al. [0027], [0035]).
PNG
media_image3.png
294
432
media_image3.png
Greyscale
The image labeling introduced by Saha et. al. is important to the claimed invention so that there is no confusion in the process of classifying a plurality of images during inspection, and is a feature of deep neural network (DNN) which is a type of machine learning model that excels in image labeling tasks. Thus, it would have been obvious to a person having ordinary skills in the art (PHOSITA) before the effective filing date of the claimed invention to have combined the teachings of Nagato et. al. with the teachings of Saha et. al. to incorporate the deep neural network ability to label images of Saha et. al. with the image processing device of Nagato et. al. that discriminates between defective and non-defective classes.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Nagato et. al. in view of Bian et. al. (US Patent 2021/0133511 A1).
Regarding claim 14, Nagato et. al. teaches the image processing device according to claim 1. However, Nagato et. al. fails to teach wherein if it is evaluated that the re-training of the machine learning model is unnecessary, after completing the training of the machine learning model, the processor inputs a plurality of evaluation images different from the plurality of training images and the plurality of validation images to the machine learning model, executes classification of classifying the plurality of evaluation images into the plurality of classes, and evaluates accuracy of the classification.
Bian et. al. teaches re-training of “tail-classes” e.g., class of images that did not meet a threshold test during inspection in order to expand the sample images for more accurate defect detection. Thus, Bian et. al. teaches wherein if it is evaluated that the re-training of the machine learning model is unnecessary, after completing the training of the machine learning model, the processor inputs a plurality of evaluation images different from the plurality of training images and the plurality of validation images to the machine learning model, the processor inputs a plurality of evaluation images different from the plurality of training images and the plurality of validation images to the machine learning model, , executes classification of classifying the plurality of evaluation images into the plurality of classes, and evaluates accuracy of the classification (classification module, Figure 3, [0017], [0018]). Thus, it would be obvious to a PHOSITA prior to the effective filing date of the claimed invention to combine the teachings of Nagato et. al. to include the techniques of Bian et. al. for expanding inspection images in order to achieve better accuracy during inspection.
PNG
media_image9.png
527
416
media_image9.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA YIFANG LIN whose telephone number is (571)272-6435. The examiner can normally be reached M-F 7:00am-6:15pm, with optional day off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JESSICA YIFANG LIN/Examiner, Art Unit 2668 July 15, 2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights