DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is responsive to remarks filed on 08/27/2025. Claims 1-15 were pending in the previous Office Action. Claims 16-20 have been added by this amendment. Applicant submits that no new matter has been added by way of this amendment. A complete response to applicants remarks follows here below.
Response to Arguments
Applicant’s arguments, see pages 7 and 8, filed 08/27/2025, with respect to the rejection(s) of claims 1, 6, 14 and 15 under 35 U.S.C. 102 have been fully considered. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Maximo (US 20200111210 A1) and Hida EP 20176620.1 using (US 20210374928 A1) as an equivalent English translation and as detailed below in the newly forwarded rejections.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim 14 is rejected under 35 U.S.C. 102(a)(1) and/or (a)(2) as being anticipated by Maximo (US 20200111210 A1).
Regarding Claim 14: (Original) Maximo discloses a method (Refer to para [001]; “The subject matter disclosed herein relates to classification of images, such as medical images, using machine learning approaches, including approaches that incorporate deep learning and/or stacked autoencoders.”) comprising: training an autoencoder model using a plurality of training images of anomaly-free physical objects of a same type (Refer to para [026]; “The technique may employ a single homogeneous group of images for training a deep-learning based autoencoder model, where images for exceptional classes are not needed for training the model. In addition, the technique does not need manual intervention in defining each class or classifying image samples in each class. Further, the images may be acquired from any equipment, such as medical-images collected using magnetic resonance or X-ray based imaging, color images collected using RGB-camera or multi-spectral images collected using specialized equipment.”) and using the autoencoder model to identify a location of an anomaly of a physical object of the same type as the anomaly-free physical objects within an extracted region of interest of a captured image of the physical object (Refer to Figures 4-8 and also para [032]; “As discussed herein, deep learning techniques (which may also be known as deep machine learning, hierarchical learning, or deep structured learning) are a branch of machine learning techniques that employ mathematical representations of data and artificial neural networks for learning and processing such representations. By way of example, deep learning approaches may be characterized by their use of one or more algorithms to extract or model high level abstractions of a type of data-of-interest. This may be accomplished using one or more processing layers, with each layer typically corresponding to a different level of abstraction and, therefore potentially employing or utilizing different aspects of the initial data or outputs of a preceding layer (i.e., a hierarchy or cascade of layers) as the target of the processes or algorithms of a given layer. In an image processing or reconstruction context, this may be characterized as different layers corresponding to the different feature levels or resolution in the data.” “Given a training data set (TDS), train a deep autoencoder model (as discussed in greater detail below) and extract and store an encoder model from the trained full autoencoder model (FIG. 4). Given the trained encoder model, encode all images in the training data set and compute and store cluster statistics characterizing about the images in the training data set (FIG. 5). Given the cluster statistics for training data set, define a procedure to compute and return the true-pixels count (TPC) of a given input image (FIG. 6). Given the true-pixels count procedure, compute and store a TPC threshold for the training data set (FIG. 7). Given the TPC threshold for the training data set, define a procedure to classify an input image in two or more classes, depending on choice of threshold values used (FIG. 8).”).
Claims 1, 4, 6, 7 and 15 are rejected under 35 U.S.C. 102(a)(1) and/or (a)(2) as being anticipated by Hida (US 20210374928 A1).
Regarding Claim 1: (Original) Hida discloses a non-transitory computer-readable data storage medium storing program code executable by a processor to perform processing (Refer to para [113]; “The memory 994 may include a computer readable medium, which term may refer to a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) configured to carry computer-executable instructions or have data structures stored thereon. Computer-executable instructions may include, for example, instructions and data accessible by and causing a general purpose computer, special purpose computer, or special purpose processing device (e.g., one or more processors) to perform one or more functions or operations. Thus, the term “computer-readable storage medium” may also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methods of the present disclosure. The term “computer-readable storage medium” may accordingly be taken to include, but not be limited to, solid-state memories, optical media and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media, including Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid state memory devices).”) comprising: extracting a region of interest from a captured image of a physical object (Refer to para [026]; “Anomalies such as random anomalies, real defects, or modified real defects, are illustrated in defect image data, and superimposed onto defect-free data (parent image) to obtain the training data images. Then, the generator neural network (autoencoder) to which the training data images are input is trained in order to remove them. In this training phase, the input to the generator neural network is defect-free images (i.e. parent images) to which defect image data such as artificial anomalies is attached (superimposed), and output is a reconstruction of the training data image without anomalies. The generator neural network which can remove anomalies such as scratches etc., is the trained via this training process.”) applying an autoencoder model to the extracted region of interest to reconstruct the region of interest (Refer to para [066]; “The combination of the autoencoder 341 and decoder 342 realizes the generator neural network 340. The decoder 342 effectively reverses the action of the autoencoder, since it takes a decoded representation of the input image and generates an image from the decoded representation. A role of the autoencoder is to extract features of images—in particular, the autoencoder is trained to extract the features of a defect-free image. The autoencoder restores input images and extracts features of input images. Skip connection such as UNet is an example of a functional part of an autoencoder..”) and identifying a location of an anomaly of the physical object within the extracted region of interest, if any, based on the extracted and reconstructed regions of interest (Refer to para [025 and 094]; “The pixel values (for example, RGB, or intensity) at equivalent pixel locations across the two versions of the image are compared and the differences assessed. The difference may be represented as a reconstruction residual, which is a difference between the acquired live image and the reconstructed image generated by the trained generator neural network 340. That is, in the live phase, a reconstruction residual (which may be referred to as a residual error map) is produced with the use only of the trained generator neural network 340. The assessment may be an aggregation across a region (i.e. a number of local pixels), or a marking of a pixel location as different or not different between the two images based on the difference in pixel value being above or below a threshold. The threshold may be predefined or may be determined adaptively based on the differences across the entire image. For example, differences more than one standard deviation from the mean difference may be flagged as different. For differences to be detected as a defect, a condition may be applied, such as, more than X % of pixels in a region of minimum size Y pixels satisfy the difference threshold. The region may be grown until the percentage threshold ceases to be satisfied. Alternatively, a defect may be an unbroken region of differences greater than a minimum pixel size.”).
Regarding Claim 4: (Original) Hida discloses applying an object segmentation model to the captured image (Refer to para [110]; “FIG. 10 shows numeric results of AUC with the worked example compared with state of the art trained generator neural networks. Prior art A is a CAE(SSIM) according to Paul Bergmann, Sindy Lowe, Michael Fauser, David Sattlegger, and Carsten Steger; Improving unsupervised defect segmentation by applying structural similarity to autoencoders; arXiv preprint:1807.02011, 2018; and prior art B is a CAE(I2) according to the same document. Prior art C is an AnoGAN according to T. Schlegl, P. Seebock, S. Waldstein, U. Schmidt-Erfurth, and G. Langs; Unsupervised anomaly detection with generative adversarial networks to guide marker discovery; International Conference on IPMI, pages 146-157; Springer, 2017. The worked example outperforms the prior art with any category from the dataset. A threshold of 5% for FPR rate is used in these results. The worked example successfully reconstructs input images without any defects, and without blurring or collapse.”).
Regarding Claim 6: (Original) Hida discloses the autoencoder model is trained on training images of anomaly-free physical objects of a same type as the physical object (Refer to para [026]; “Anomalies such as random anomalies, real defects, or modified real defects, are illustrated in defect image data, and superimposed onto defect-free data (parent image) to obtain the training data images. Then, the generator neural network (autoencoder) to which the training data images are input is trained in order to remove them. In this training phase, the input to the generator neural network is defect-free images (i.e. parent images) to which defect image data such as artificial anomalies is attached (superimposed), and output is a reconstruction of the training data image without anomalies. The generator neural network which can remove anomalies such as scratches etc., is the trained via this training process.”)
Regarding Claim 7: (Original) Hida discloses generating a residual map between the extracted region of interest and the reconstructed region of interest (Refer to para [108]; “As an output of the image comparer 620 in the live phase, a residual map is computed when calculating the difference between an input and a reconstructed image. Blurring with window size of 11 is used to get rid of noise. Finally, Area Under the Curve (AUC) is computed by calculating True Positive Rate (TPR), False Positive Rate (FPR), and Receiver Operating Characteristics (ROC) curve with pixel-wise thresholds of 0-255 range.”) and removing any pixel of the residual map having a value less than a threshold (Refer to para [063-065]; “The defect image data is images of anomalies or defects that may be attached to the set of images of defect-free physical samples at S202. The defect image data may include zero, less than 10, or less than 20% (in area) image of defect-free physical samples (i.e. in some implementations defect image data may incorporate a small amount of defect-free image). The training data image manager 330 accesses the stored set of images of defect-free physical samples from the first storage location 310, and defect image data from the second storage location 320 (when required), to produce training data images for input to the generator neural network 340. The training data image manager 330 may control one or more from among number, transparency, color, size, and shape, of defect image data to attach to the parent images, in order to cause an amount of difference between the parent image and the training data image, which amount of difference is controlled according to a training condition such as number of completed training epochs. The generator neural network 340 comprises an encoder 341 and a decoder 342. The generator neural network 340 may be an autoencoder comprising an encoder 341 and a decoder 342. Specifically, it may be a convolutional autoencoder. The convolutional autoencoder has a multilayer neural network architecture comprising one or more convolutional layers, followed by sub-sampling layers (pooling) and one or more fully connected layers. An example is a skip connection such as UNet (https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/).”).
Regarding Claim 15: (Original) Hida discloses a computing device comprising: a processor (Refer to para [111]; “The computing device comprises a processor 993, and memory, 994”) and a memory storing instructions executable by the processor to (Refer to para [113]; “The memory 994 may include a computer readable medium, which term may refer to a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) configured to carry computer-executable instructions or have data structures stored thereon.”): preprocess an image of a physical object to crop the image (Refer to para [090]; “The acquiring step S102 may include preprocessing of the image to a size and/or format required by the generator neural network 340.”) apply an autoencoder model to the preprocessed image to generate a reconstructed image (Refer to para [066]; “The combination of the autoencoder 341 and decoder 342 realizes the generator neural network 340. The decoder 342 effectively reverses the action of the autoencoder, since it takes a decoded representation of the input image and generates an image from the decoded representation. A role of the autoencoder is to extract features of images—in particular, the autoencoder is trained to extract the features of a defect-free image. The autoencoder restores input images and extracts features of input images. Skip connection such as UNet is an example of a functional part of an autoencoder.”) and identify a location of an anomaly of the physical object within the image (Refer to para [025 and 094]; “The pixel values (for example, RGB, or intensity) at equivalent pixel locations across the two versions of the image are compared and the differences assessed. The difference may be represented as a reconstruction residual, which is a difference between the acquired live image and the reconstructed image generated by the trained generator neural network 340. That is, in the live phase, a reconstruction residual (which may be referred to as a residual error map) is produced with the use only of the trained generator neural network 340. The assessment may be an aggregation across a region (i.e. a number of local pixels), or a marking of a pixel location as different or not different between the two images based on the difference in pixel value being above or below a threshold. The threshold may be predefined or may be determined adaptively based on the differences across the entire image. For example, differences more than one standard deviation from the mean difference may be flagged as different. For differences to be detected as a defect, a condition may be applied, such as, more than X % of pixels in a region of minimum size Y pixels satisfy the difference threshold. The region may be grown until the percentage threshold ceases to be satisfied. Alternatively, a defect may be an unbroken region of differences greater than a minimum pixel size.”) if any, based on a residual map between the preprocessed image and the reconstructed image (Refer to para [108]; “As an output of the image comparer 620 in the live phase, a residual map is computed when calculating the difference between an input and a reconstructed image. Blurring with window size of 11 is used to get rid of noise. Finally, Area Under the Curve (AUC) is computed by calculating True Positive Rate (TPR), False Positive Rate (FPR), and Receiver Operating Characteristics (ROC) curve with pixel-wise thresholds of 0-255 range.”).
Regarding Claim 19: (New) Hida discloses a extracting a region of interest from the image of the physical object (Refer to para [026]; “Anomalies such as random anomalies, real defects, or modified real defects, are illustrated in defect image data, and superimposed onto defect-free data (parent image) to obtain the training data images. Then, the generator neural network (autoencoder) to which the training data images are input is trained in order to remove them. In this training phase, the input to the generator neural network is defect-free images (i.e. parent images) to which defect image data such as artificial anomalies is attached (superimposed), and output is a reconstruction of the training data image without anomalies. The generator neural network which can remove anomalies such as scratches etc., is the trained via this training process.”).
Regarding Claim 20: (New) Hida discloses applying an object segmentation model to the image of the physical object (Refer to para [110]; “FIG. 10 shows numeric results of AUC with the worked example compared with state of the art trained generator neural networks. Prior art A is a CAE(SSIM) according to Paul Bergmann, Sindy Lowe, Michael Fauser, David Sattlegger, and Carsten Steger; Improving unsupervised defect segmentation by applying structural similarity to autoencoders; arXiv preprint:1807.02011, 2018; and prior art B is a CAE(I2) according to the same document. Prior art C is an AnoGAN according to T. Schlegl, P. Seebock, S. Waldstein, U. Schmidt-Erfurth, and G. Langs; Unsupervised anomaly detection with generative adversarial networks to guide marker discovery; International Conference on IPMI, pages 146-157; Springer, 2017. The worked example outperforms the prior art with any category from the dataset. A threshold of 5% for FPR rate is used in these results. The worked example successfully reconstructs input images without any defects, and without blurring or collapse.”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Maximo (US 20200111210 A1) and in combination with Hida (US 20210374928 A1).
Regarding Claim 16: (New) Maximo discloses all the claimed elements as rejected above. Maximo does not expressly disclose pixel removal based on a threshold value.
Hida teaches “machine learning algorithms to automate defect detection and identification.”
Hida discloses generating a residual map between the extracted region of interest and a reconstructed region of interest generated by the autoencoder model based on the captured image (Refer to para [108]; “As an output of the image comparer 620 in the live phase, a residual map is computed when calculating the difference between an input and a reconstructed image. Blurring with window size of 11 is used to get rid of noise. Finally, Area Under the Curve (AUC) is computed by calculating True Positive Rate (TPR), False Positive Rate (FPR), and Receiver Operating Characteristics (ROC) curve with pixel-wise thresholds of 0-255 range.”) and removing any pixel of the residual map having a value less than a threshold (Refer to para [063-065]; “The defect image data is images of anomalies or defects that may be attached to the set of images of defect-free physical samples at S202. The defect image data may include zero, less than 10, or less than 20% (in area) image of defect-free physical samples (i.e. in some implementations defect image data may incorporate a small amount of defect-free image). The training data image manager 330 accesses the stored set of images of defect-free physical samples from the first storage location 310, and defect image data from the second storage location 320 (when required), to produce training data images for input to the generator neural network 340. The training data image manager 330 may control one or more from among number, transparency, color, size, and shape, of defect image data to attach to the parent images, in order to cause an amount of difference between the parent image and the training data image, which amount of difference is controlled according to a training condition such as number of completed training epochs. The generator neural network 340 comprises an encoder 341 and a decoder 342. The generator neural network 340 may be an autoencoder comprising an encoder 341 and a decoder 342. Specifically, it may be a convolutional autoencoder. The convolutional autoencoder has a multilayer neural network architecture comprising one or more convolutional layers, followed by sub-sampling layers (pooling) and one or more fully connected layers. An example is a skip connection such as UNet (https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/).”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Maximo by adding a “neural network to generate defect-free reconstructions of input images; acquiring a live image of the subject physical sample; inputting the live image to the trained generator neural network to generate a defect-free reconstruction of the live image; comparing the defect-free reconstruction of the live image with the acquired live image to determine a difference; and identifying a defect corresponding to the subject physical sample at a location of the determined difference…” as taught by Hida.
The suggestion/motivation for combining the teachings of Maximo and Hida would have been in order to “train a generator neural network to produce defect-free or reduced-defect versions of images of physical samples, which, upon comparison with the actual images, reveal the locations of defects. In this manner, AI-based defect identification can be performed in the absence of labelled training data.” (at para [024]; Hida).
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Maximo and Hida in order to obtain the specified claimed elements of Claim 16. It is for at least the aforementioned reasons that the Examiner has reached a conclusion of obviousness with respect to the claim in question.
Allowable Subject Matter
Claims 2, 3, 8, 9, 10 11, 12, 13, 17 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The prior art either singly or in combination does not teach, disclose or suggest at least the following claim limitation(s): “… aligning the captured image of the physical object against a reference image of another physical object of a same type as the physical object; and cropping the aligned captured image based on a bounding box identifying a corresponding region of interest within the reference image, wherein the cropped aligned captured image constitutes the region of interest and wherein aligning the captured image against the reference image comprises calculating a transformation matrix that aligns the captured image to the reference image, and wherein cropping the aligned captured image based on the bounding box comprises applying an inverse of the transformation matrix to the bounding box and cropping the captured image using the inverse-applied bounding box.”
Similarly, the prior art either singly or in combination does not teach, disclose or suggest at least the following claim limitation(s): “… wherein using the autoencoder model to identify the location of the anomaly of the physical object within the extracted region of interest further comprises: after removing any pixel having a value less the threshold, applying a morphological operation to the residual map.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MIA M THOMAS whose telephone number is (571)270-1583. The examiner can normally be reached M-Th 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen (Steve) Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MIA M. THOMAS
Primary Examiner
Art Unit 2665
/MIA M THOMAS/Primary Examiner
Art Unit 2665