DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
2. This communication is in response to the application filled on 03/27/2024.
3. Claims 1-18 are pending.
4. Limitations appearing inside {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Information Disclosure Statement
5. The information disclosure statements (IDS) submitted on 04/08/2025 and 08/20/2025 have been considered by the examiner.
Claim Objections
6. Claims 1, 7, and 13 objected to because of the following informalities:
In ln. 7, claim 1 recites “…used to train to obtain a defect detection model…”, consider changing to recite “…used to train a defect detection model…”. Claims 7 and 13 recite analogous language to claim 1 in ln. 12 and 8 respectively.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 1-4, 7-10, 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over CN-110175659-A to Scott et al. (hereinafter Scott) in view of JP-2017054239-A to Matsumura (hereinafter Matsumura).
10. Regarding Claim 1, Scott discloses a method for processing sample data, comprising ([pg. 5, ln. 11-14] “…spinning machine fault monitoring method provided by the invention can be used in system spinning machine, the spinning machine system relates to a server and a camera, server obtains the camera spinning pre-set image area collecting and monitoring the image, thereby determining whether the spinning machine goes wrong.”):
obtaining a plurality of first working images of a spinning box ([pg. 5, ln. 19-39] “S102, the image of the predetermined area to be monitoring obtaining of the spinning machine… wherein the spinning machine is a machine for forming the polymer solution or melt forming a filament, which is for spinning device of terylene, chinlon and polypropylene, etc., a predetermined area of the spinning machine can be spinning window, also can be other position of the spinning machine, such as downstream of the outlet wire and so on… in a spinning machine is provided with a spinning window for observing the operation condition of spinning machine wire feed nozzle, and the quality of the wire, through the spinning window can observe the number of filament finally condensed into a line, and can observe the wire fault in the spinning operation in, for example, the wire is uneven, discontinuous, emitting filaments and silk error entanglement and failure. in a spinning machine can comprises a plurality of the wire nozzle; correspondingly, each nozzle is provided with a spinning window, convenient to observe the state of the spinning operation… the predetermined area selecting spinning window as spinning machine, obtaining the image of the spinning window. Specifically, setting the camera in front of each of the spinning window, for real-time image of the spinning window real-time obtaining spinning in the running process. real-time image of the spinning window for obtaining can be transmitted to the server, after the server obtaining the real-time image, capable of pre-processing the real-time image so as to obtain the pre-processed image, as an image to be monitored.”);
for each first working image in the plurality of first working images, determining the first working image as a defect image of the spinning box when {a difference between} the first working image and a normal image of the spinning box meets a first condition ([pg. 8, ln. 24-33] “…the failure prediction model to be of image input to training before the forecasting time after using the failure prediction model after training to be predicted before the time of image for prediction, determining spinning machine in time is to be predicted will fault. the fault prediction model comprises a first network, a second network… the first network may be a convolutional neural network, the second network may be a deep learning model…”, [pg. 8, ln. 34-55] “… the fault prediction model training method of training comprises: after obtaining the predetermined area of spinning machine training sample of historical image, said historical image training sample comprises: history image before the failure occurs and the corresponding fault occurrence identifier, using the history image training sample, training the failure prediction model established in advance to obtain the fault prediction model after training… firstly establishing an initial fault prediction model according to the historical fault information of the spinning machine, the historical image before the failure occurs as the input, the corresponding fault occurrence identifier as output, the initial fault prediction model obtained by training a supervised, fault prediction model after training. the fault identification spinning opportunity for representing the fault. the historical fault information can be uploaded to the server by the worker, the image history fault information may include spinning fault when a predetermined region of the machine output to fault, and parameters and the like… in the process of acquiring the history image of the before failure of the history image in the training sample, the server can store the real-time image of the predetermined area collected by the camera in the spinning machine of spinning machine when a fault occurs, storing the obtained from the image before the fault history image for a period of time. the obtaining process of the historical image before the failure occurs”, [pg. 9, ln. 1-13] “…history image training samples further comprises: random time not having fault for the history image and the corresponding fault not occurrence identifier. fault does not mark for representing the spinning machine will not fault. Specifically, during the training process of the fault prediction model, the history image of random time may also be a failure is not occurring as input, the corresponding identifier as an output failure has not occurred, the fault prediction model for training. not generating negative sample identification corresponding to the training model of non-fault history image and the corresponding fault, formal fault prediction model for fault prediction before, can use a plurality of different negative samples for training so as to improve the accuracy of fault prediction… the history image prediction spinning machine according to the predetermined area of the spinning machine whether occurs fault at some future time, so as to make the machine maintenance work, maintenance personnel can advance maintenance spinning machine of the fault in time and reduces the loss caused by fault.”); and
obtaining a first sample data set based on all defect images in the plurality of first working images; wherein the first sample data set is used to train to obtain a defect detection model of the spinning box ([pg. 8, ln. 24-33], [pg. 8, ln. 34-55], [pg. 9, ln. 1-13]), and the defect detection model is used to detect whether a second working image of the spinning box has a defect ([pg. 8, ln. 24-33], [pg. 8, ln. 34-55], [pg. 9, ln. 1-13]).
One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the spinning window of Scott to be analogous to a “spinning box”. Scott does not specifically disclose determining the first working image as a defect image based on a difference between the first working image and a normal image, though one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that Scott determines there is a defect when a first condition between a first working image and a normal image is met. Specifically, Scott discloses wherein the first condition to determine a defect image is if the fault prediction model determines there is a defect after training using a normal and first working image as training parameters.
However, Matsumura teaches wherein determining the first working image as a defect image when a difference between the first working image and a normal image meets a first condition ([par. 0022, ln. 1-5] “The defect detection section 41 compares the captured image of the substrate 9 with a reference image showing the same region (normal region) as the captured image to obtain a difference image (typically, the absolute value of the difference between both images is), And a defect which is an abnormal part is detected based on the difference image. Then, a defect image which is a multivalued image of the defect portion is generated. In the defect detection unit 41, defects may be detected by other methods.”, [par. 0030, ln. 1-8] “…defect detection unit 41 compares the captured image of the substrate 9 with the reference image, thereby obtaining a difference image, and a defect image is generated based on the difference image. At this time, in addition to the defect image, a portion indicating the same area as the defect image in the reference image and the difference image (hereinafter simply referred to as "reference image" and "difference image") and a binary image indicating the defect area… An image obtained by binarizing the differential image, hereinafter referred to as "mask image") is also generated. In each teacher data set 81, in addition to the teacher image, a reference image, a difference image and a mask image corresponding to the teacher image are included.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Scott and Matsumura as within the same field of industrial image inspection for defect detection using normal images, and as analogous to the claimed invention. The motivation to combine would have been obvious to one of ordinary skill in the art, and is disclosed in Matsumura, wherein it improves accuracy of the trained model ([par. 0017, ln. 1-3] “…it is possible to construct a classifier specialized for each attribute item by using a sufficient number of teacher data sets, and as a result, classification performance can be improved.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Scott with the difference-based defect detection of Matsumura through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Scott with the difference-based defect detection of Matsumura to obtain the invention as specified in claim 1.
11. Regarding Claim 2, a combination of Scott and Matsumura teaches the method of claim 1. Scott does not specifically disclose classifying all the defect images according to at least one defect type, to obtain at least one image set corresponding to the at least one defect type one by one; and obtaining the first sample data set based on the at least one image set.
However, Matsumura teaches classifying all the defect images according to at least one defect type, to obtain at least one image set corresponding to the at least one defect type one by one; and obtaining the first sample data set based on the at least one image set ([par. 0029, ln. 5-8] “Teaching information on a plurality of setting attribute items for each defect image and the defect image (that is, a defect image on which a plurality of setting attribute items have been taught, hereinafter referred to as "teacher image") are mutual And stored in the teacher data storage unit 331 as the teacher data set 81. In this way, a plurality of teacher data sets 81 are prepared (step S 11).”, [par. 0032, ln. 1-5] “Subsequently, a plurality of classifiers 324 that individually classify the plurality of setting attribute items are individually constructed by the classifier building unit 33 (step S 12). Here, each classifier 324 classifies images by paying attention only to the setting attribute items associated with the classifier 324. In the construction of each classifier 324, the feature amount used in the classifier 324 is acquired from the image included in the teacher data set 81 by the feature amount calculation unit 323.”, [par. 0033, ln. 1-8] “in the classifier 324 for classifying the type of the defect, geometric feature quantities (here, feature quantity vectors which are a set of a plurality of feature quantities) such as the area of the defect region, the perimeter, the gravity center position, etc. are used. Therefore, the geometrical feature amount is acquired based on the defect region indicated by the mask image 73 included in each teacher data set 81. Then, the learning unit 335 learns the classifier 324 using the teaching information of the type of the defect in all the teacher data sets 81 and the geometrical feature amount, thereby performing classification on the type of the defect A classifier 324 is constructed. In the classifier 324, it is possible to output classification results of foreign matter, defects, bubbles in a classification operation to be described later.”, [par. 0041, ln. 1-6] “Here, a classification unit of a comparative example in which the nine defect classes are classified by one classifier is assumed. In the construction of a classifier (multi-class classifier) in the classification unit of the comparative example, a combination of teaching information on a plurality of setting attribute items included in each teacher data set is regarded as "teaching defect class", and in the teacher data set A classifier that classifies defect classes is constructed by learning the classifier using the teaching defect class, the texture feature quantity, and the geometric feature quantity.”). The motivation to combine remains analogous to claim 1, specifically, in that classifying the defect images into image sets comprising at least one at least one defect type one by one can improve the accuracy of the trained model ([par. 0017, ln. 1-3], [par. 0047, ln. 3-5] “Therefore, it is possible to construct the classifier 324 specialized for each setting attribute item by using a sufficient number of teacher data sets 81, and as a result, the classification performance can be improved.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Scott with the difference-based defect detection and dataset/defect classification of Matsumura through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Scott with the difference-based defect detection and dataset/defect classification of Matsumura to obtain the invention as specified in claim 2.
12. Regarding Claim 3, a combination of Scott and Matsumura teaches the method of claim 2. Scott discloses wherein the at least one defect type comprises at least one of defects of a spinneret, a yarn, a yarn guide hook and an oil nozzle in the spinning box ([pg. 6, ln. 37-46] “…firstly establishing an initial fault identification model according to the fault type of the spinning machine, in order to improve the fault identification model distinguishing ability of fault type, the input pre marked spinning operation image with a fault type of the fault, the initial fault identification model obtained by training a supervised, fault identification model after training. wherein, pre-marking with spinning operation image of the fault type of the fault, i.e., image training sample spinning operation, are derived from image of the spinning window of spinning machine when a fault occurs, in the spinning window with nozzle out wire through the spinning window can be observed a lot of filaments finally condensed into line. the fault type of the mark, i.e. the fault type mark, used for indicating the fault of generating belongs to which type, may be, for example, a wire discharging uneven, discontinuous, emitting filaments and silk error entanglement and failure type.”). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Scott with the difference-based defect detection and dataset/defect classification of Matsumura to obtain the invention as specified in claim 3.
13. Regarding Claim 4, a combination of Scott and Matsumura teaches the method of claim 2. Scott teaches wherein for each image set in the at least one image set, clipping each image in the image set according to a defect {type corresponding to the image set}, to obtain a detection area image set corresponding to the image set; and obtaining the first sample data set based on at least one detection area image set corresponding to the at least one image set one by one ([pg. 5, ln. 9-43] “…FIG. 2, the image of the predetermined area to be monitoring for obtaining spinning machine, specifically comprising the following step S1021 to step S1023. S1021, real-time image of the predetermined area of the obtained spinning machine. Specifically… set in a predetermined area of a camera module, a real-time image of the predetermined region of the spinning machine server obtains camera module collecting. the camera module can be formed by multiple cameras, for real-time image collecting spinning machine running when working… collecting the real time image of nozzle-out wire in the spinning window… real time image can be spinning predetermined area of the collected image according to the predetermined time interval, for example, may be every 3 minutes, each collecting a piece of image of a predetermined region, or each every 3 minutes after collecting the image for a period of time, such as video image acquisition for 30 seconds, this is not limited. S1022. The preset frame number intervals and/or time intervals, and frame sampling the real-time image to obtain at least one frame sampled image. Specifically, the frame sampling the real-time image processing, can be the preset time interval extraction real-time image predetermined number of image frames, such as in real-time image every 20 second, extracting 2 image frame, also can be the preset frame number interval extraction real-time image predetermined number of image frames, such as in a real-time image every 100 frame extracting 2 image frame. process of the frame sampling can be realized using an algorithm or application program, such as real time image can be set in the server application, using the application program sent by the camera to frame sampling. algorithm or application program of the frame sampling also can be set in the camera, the camera collects real-time image, the frame sampling using the algorithm or application program in real time. S1023, the at least one frame sampled image splicing according to the preset rule, obtaining the to-be monitored image of the predetermined area. Specifically, the frame sampling to obtain sampling at least one frame image, can also be for splicing processing for at least one frame sampled image using an algorithm or application program according to the preset rule, generating a preprocessed image, as the image to be monitored. wherein the preset rule is the image splicing rules, according to certain rules for the plurality of images into one image, and may be, for example, the 16 image according to the horizontal matrix distributed spliced into one image, so that a plurality of frames sampled image after performing splicing processing can be generated with less image to be monitored. the splicing algorithm or application program can be set in the camera or server, not as defined herein. by sampling the frame and image pre-processing method for image splicing, it can effectively reduce the time of the convolution image by convolution neural network, thereby increasing the efficiency of fault monitoring”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize the image sets of Scott to be analogous to the clipping the detection area image set corresponding to the defect, given the broadest reasonable interpretation of “clipping” including clipping video/frames as disclosed in Scott, but that Scott does not specifically disclose wherein this clipping is performed according to a defect type. If a narrower interpretation of “clipping” is taken to only encompass removing the parts of an image as opposed to removing the image from a set of images (e.g., background removal, see Specifications par. [0050]), the examiner notes Scott would not read on the claimed invention, but that such clipping is typical pre-processing and that Scott discloses pre-processing ([pg. 8, ln. 18-21] “the predetermined area of the obtained spinning machine after the image to be predicted before the current time, pre-processing the image, sampling pre-processing comprises frame and splicing processing, frame sampling and specific way splicing processing may refer to the above embodiments, which will not be repeated here.”), and as such would likely be obvious in view of other references of record (see PTO-892). Therefore, given the broadest reasonable interpretation of clipping, Scott discloses clipping each image in the image set according to a defect to obtain a detection area image set, but does not specifically disclose wherein this is performed according to a type of defect corresponding to the image set.
However, Matsumura teaches wherein the image sets correspond to the at least one defect type one by one ([par. 0029, ln. 5-8], [par. 0032, ln. 1-5], [par. 0033, ln. 1-8], [par. 0041, ln. 1-6]). The motivation to combine remains analogous to claim 2, wherein training datasets comprises of defect type specific samples result in improved model performance ([par. 0017, ln. 1-3], [par. 0047, ln. 3-5]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that different defects benefit from alternative clipping methodologies ([pg. 6, ln. 37-46] e.g., discontinuity would be immediately recognizable in an image due to no yarn being present/no continuity between yarn sections, whereas uneven discharging and/or entanglement would likely require continuous clipping for a period of time to determine a nozzle is distributing more than another), and subsequently modified the clipping of Scott to correspond to at least one defect type analogous to the sets disclosed in Matsumura to improve model accuracy. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Scott with the difference-based defect detection and dataset/defect classification of Matsumura through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Scott with the difference-based defect detection and dataset/defect classification of Matsumura to obtain the invention as specified in claim 4.
14. Regarding Claim 7, the claim language is analogous to claim 1, with the exception of “An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores an instruction executable by the at least one processor, and the instruction, when executed by the at least one processor, enables the at least one processor to execute:” wherein the remainder of the claim is analogous to claim 1. Scott discloses an electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores an instruction executable by the at least one processor, and the instruction, when executed by the at least one processor, enables the at least one processor to execute their method ([pg. 10, ln. 45-53] “…there is provided a computer device, the computer device may be a server, the internal structure can be as shown in FIG. 7. the computer device includes a processor, a memory and a network interface connected via a system bus. wherein the processor of the computer apparatus for providing calculation and control capability. the memory of the computer device comprises a non-volatile storing medium, an internal memory. the non-volatile storage medium stored with operating system and computer program. operation of the operating system and the computer program in the memory is a non-volatile storing medium provided in the environment. the network interface of the computer device is used for connecting the communication through the network with the external terminal. the computer program when executed by a processor for carrying out a fault monitoring method for spinning.”). Rejections analogous to claim 1 are further applicable to the remainder of claim 7. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the electronic device of Scott with the difference-based defect detection of Matsumura to obtain the invention as specified in claim 7.
15. Regarding Claim 8-10, a combination of Scott and Matsumura teaches the electronic device of claim 7. The claim language of claims 8-10 is analogous to claims 2-4 respectively. Rejections analogous to claims 2-4 are further applicable to claims 8-10 in view of the electronic device of Scott. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the electronic device of Scott with the difference-based defect detection and dataset/defect classification of Matsumura to obtain the invention as specified in claims 8-10.
16. Regarding Claim 13, the claim language is analogous to claim 1, with the exception of “A non-transitory computer-readable storage medium storing a computer instruction thereon, wherein the computer instruction is used to cause a computer to execute:”, wherein the remainder of the claim is analogous to claim 1. Scott discloses A non-transitory computer-readable storage medium storing a computer instruction thereon, wherein the computer instruction is used to cause a computer to execute their method ([pg. 10, ln. 45-53]). Rejections analogous to claim 1 are further applicable to the remainder of claim 13. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the non-transitory computer-readable storage medium of Scott with the difference-based defect detection of Matsumura to obtain the invention as specified in claim 13.
17. Regarding Claim 14-16, a combination of Scott and Matsumura teaches the non-transitory computer-readable storage medium of claim 13. The claim language of claims 14-16 is analogous to claims 2-4 respectively. Rejections analogous to claims 2-4 are further applicable to claims 14-16 in view of the electronic device of Scott. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the non-transitory computer-readable storage medium of Scott with the difference-based defect detection and dataset/defect classification of Matsumura to obtain the invention as specified in claims 14-16.
18. Claim 5, 11, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over CN-110175659-A to Scott, in view of JP-2017054239-A to Matsumura, and further in view of “ICNN: An Iterative Implementation of Convolutional Neural Networks to Enable Energy and Computational Complexity Aware Dynamic Approximation” to Neshatpour et al. (hereinafter Neshatpour).
19. Regarding Claim 5, a combination Scott and Matsumura teaches the method of claim 1. Scott discloses {determining a second sample data set} from the first sample data set based on a feature quantity of each image in the first sample data set {when a confidence of the defect detection model does not meet a second condition}; and updating the defect detection model based on the {second} sample data set ([pg. 4, ln. 40 to pg. 5, ln. 8] “S104, the image to be monitored for convolution to generate spinning feature map of the predetermined area. Specifically, after the image to be monitored to obtain spinning window, the image to be monitored can be input to the fault monitoring model established in advance, the fault monitoring model comprises a first network, a second network, an input end and an output end, wherein, the first network may be a convolutional neural network, the second network may be a deep learning model. the input end of the fault monitoring model connected convolutional neural network, the output end is connected with deep learning model… performing convolution using a convolutional neural network to be monitoring image, extracting the characteristic of the monitoring image to obtain the spinning feature map of the spinning window. wherein, the convolutional neural network can be provided with multiple, each convolutional neural network receiving the corresponding to-be-detected image so as to improve the efficiency of image convolution extract features. S106, the spinning feature map for monitoring, determining whether there is fault in the spinning machine. Specifically, after obtaining the spinning feature map of the spinning window, monitoring the spinning feature map using deep learning model, judging whether there is fault in the spinning machine. wherein, in the process of spinning operation, due to the low efficiency of fault occurs, so the deep learning model can be used after establishing, in the using process input normal image a lot of data and its output is the same, when the data output is not accordant, may indicate that there is failure.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that Scott discloses wherein feature quantities of the first sample data set are used to update the learning model, but does not specifically disclose the creation of a second sample dataset separate from the first sample dataset when a confidence of the defect detection model does not meet a second condition. Likewise, Matsumura does not specifically disclose determining a second sample data set from the first sample data set based on a feature quantity of each image in the first sample data set when a confidence of the defect detection model does not meet a second condition.
However, Neshatpour specifically discloses determining a second sample data set from the first sample data set based on a feature quantity of each image in the first sample data set when a confidence of the defect detection model does not meet a second condition ([pg. 552, Fig 3], [pg. 553, col. 1, IV. Iterative Learning, par. 1, ln. 7 to col. 2, par. 2, ln. 9] “Our proposed reformulation is driven by the needs of resource-constrained vision applications for lowering the energy consumption and shortening the classification latency when deploying CNN solutions. In the proposed solution, a large CNN block is decomposed into many smaller networks (u-CNN in Figure 1), allowing iterative refinement and greater control over the execution of algorithm. Thus, not all images pass through all the u-CNNs; By monitoring the successive execution of u-CNN networks, a thresholding mechanism decides when to terminate the forward u-CNN traversal based on the current classification confidence of the images. The proposed solution requires sub-sampling of input images into various sets for various rounds of computation. We propose the application of Discrete Wavelet sampling to decompose an input image into various input sets (sub-bands). The learning is then initiated using the first sub-sampled input set. Upon completion of first computational round (first u-CNN), the classification confidence is tested. If the confidence is unsatisfactory, it could be progressively increased by working on additional input samples (chosen from remaining sub-bands). Discrete Wavelet Transformation (DWT) provides the proposed learning algorithm with an attractive start point and unlike Fourier transform, in addition to frequency information, it also preserves temporal information of an image[13]… A high-level representation of envisioned iterative learning algorithm fed by DWT is illustrated in Figure 3. Each iteration is a u-CNN, which takes a new DWT sub-band as its input and refines the confidence of learning network. DWT, being a convolutional filter, could be readily computed using processing elements (PE) in CNN processing engine of interest, or could be provided directly to CNN. The iterative transformation of learning algorithm has many advantages: It could be terminated as soon as a u-CNN produces the desired confidence level. Further iterations could be avoided if the first u-CNN detection confidence is below a certain threshold signifying no contextually significant input. And confidence could be improved by moving to the next iteration, if the current measure of confidence remains between demarcated thresholds, aiding the rise or decline of classification confidence.”, [pg. 555, col. 1, VIII. Results, par. 3, ln. 1 to pg. 555, col. 2, par. 1, ln. 8] “To exploit the iterative CNN for reducing computational complexity, for each image in the validation set (50K images), ICNN starts the classification with the first iteration and calculates the classification confidence. Classification confidence is calculated by adding up the probabilities of the top-5 classes. Subsequently, the classification confidence is compared to the desired Confidence Threshold (Ct ), and if lower, ICNN moves forward to the next iteration. ICNN continues this process until the classification confidence is higher that Ct. In this approach, various images are detected in different iterations. Moreover, some of the images never reach a classification confidence above Ct. For these images, the results of the last iteration are used…”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Scott, Matsumura, and Neshatpour as within the same field of image inspection using feed-forward neural networks, and as analogous to the claimed invention. The motivation to combine is disclosed in Nesthapour, wherein it significantly improves processing performance while maintaining comparable accuracy and allows for fine-tuned model parameters ([pg. 554, Fig. 6], [pg. 554, col. 2, VII. Complexity Analysis, par. 1, ln. 1-14] “As mentioned in the previous section, the number of Ofmaps in each u-CNN is considerably reduced with respect to the original CNN. Figure 6 shows the number of Floating Point Operations (FLOP) for a forward pass of the iterative AlexNet per image at each iteration. The figure shows that, even if executed to the last iteration, ICNN still has a lower computational complexity (needing 30% fewer FLOPs) than the original AlexNet. On top of this, many images are detected at earlier iterations, removing the need to process subsequent u-CNNs. This further reduces the total FLOP count for a large number of images. More specifically, images detected in iterations 1, 2, 3, 4, 5 and 6 respectively require 12.2×, 6.1×, 4×, 3×, 2.3× and 1.8× fewer FLOPs when compared to the original AlexNet.”, [pg. 555, col. 2, par. 2, ln. 1 to pg. 556, col. 1, par. 1, ln. 9] “Figure 8 shows that with each new iteration, the ICNN detects more images with high classification confidence. This is to be expected, as the last iterations combine a larger number of features in more complex architectures with a larger number of parameters, allowing a more accurate prediction model and thus classifying more images. Moreover, by increasing the value of Ct, the number of images classified in early iterations decreases; however the classification accuracy (correct label within top-5) increases. In Figure 8 this is illustrated by comparing the difference in the heights of Top-5 and Detected bars at each iteration, where a larger delta means larger miss-classification. More specifically, higher values of Ct enhances the accuracy at the expense of larger computation complexity, and lower Ct values reduce the complexity at the expense of lower classification accuracy. Thus, an intelligent selection of Ct maintains a trade-off between accuracy and computational complexity. Note that the Ct doesn’t have to be a fixed value across different u-CNN iterations and could be different for each iteration. In addition, it could be tuned at run-time to dynamically control the trade-off between accuracy and computational complexity. Access to such run-time control knob is extremely desirable and is a new concept in CNN networks which is made possible by ICNN. Figure 9 illustrates this trade-off where the overall accuracy of ICNN, as well as the average number of FLOPs required to process the 50K images in the validation-set changes with the selection of Ct values (Ct is fixed across all u-CNN layers). Interestingly, with a fixed confidence threshold of 0.9, the overall accuracy is the same as using the data from all the iterations to process all images (see Figure 7) while requiring only half the FLOPs.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the method of Scott with the difference-based defect detection of Matsumura, and further combined the method of the combination of Scott and Matsumura with the second sample data set determination based on confidence as disclosed in Neshatpour, through known means with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the method of Scott with the difference-based defect detection of Matsumura and the second sample data set determination based on confidence as disclosed in Neshatpour to obtain the invention as specified in claim 5.
20. Regarding Claim 11, a combination of Scott and Matsumura teaches the electronic device of claim 7. Rejections analogous to claim 5 are further applicable to claim 11 in view of the electronic device of Scott. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the electronic device of Scott with the difference-based defect detection of Matsumura and the second sample data set determination based on confidence as disclosed in Neshatpour to obtain the invention as specified in claim 11.
21. Regarding Claim 17, a combination of Scott and Matsumura teaches the non-transitory computer-readable storage medium of claim 13. Rejections analogous to claim 5 are further applicable to claim 17 in view of the electronic device of Scott. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the non-transitory computer-readable storage medium of Scott with the difference-based defect detection of Matsumura and the second sample data set determination based on confidence as disclosed in Neshatpour to obtain the invention as specified in claim 17.
Allowable Subject Matter
22. Claims 6, 12, and 18 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
23. The following is a statement of reasons for the indication of allowable subject matter:
While references of record discuss clustering as a means to improve model performance, the examiner specifically notes that there is a distinct difference between the clustering as performed in Neshatpour and the clustering as described in claims 6, 12, and 18, in that a target image is selected based on a feature quantity of the image and subsequently used to obtain the second data set. This is a distinction over references of record, since the clustering as performed in Neshatpour is not for the purpose of selecting a target image from a set of images, but rather to classify the image, and thus while Neshatpour would read on determining a second data set based on feature quantities when a confidence does not meet a second condition, it fails to teach or suggest obtaining a second sample data set based on the target image in each cluster. The examiner specifically notes that in the case of Neshatpour, the feature quantities are effectively variations of the same image, and that the determination of the second sample data set thus effectively comprises said features as determined by transforms of the original image such that the number of subsbands expands as the model iterates, whereas the language of claims 6, 12, and 18 lend themselves to a single selection of a target image from a clustering of images, which Neshatpour actively teaches against since this would be a reduction of the number of subsbands as opposed to an expansion.
Conclusion
24. The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. See PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAULO ANDRES GARCIA whose telephone number is (703)756-5493. The examiner can normally be reached Mon-Fri, 8-4:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached on (571)272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAULO ANDRES GARCIA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669