DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) was submitted on 9/22/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 7-10 are objected to because of the following informalities:
Claims 2-4, 16, and 17, “buffer data” is not associated with the cumulative average SNNL in claim 1. Examiner suggests amending to “buffer data set” for clarity.
Claim 10, the “third label” appears to be referencing the same part as the “first label” since both “match the largest number”. Examiner suggests amending to make clear the difference between the two labels.
Appropriate correction is required.
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 10 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 10, the “third label” is only mentioned in ¶0025 and ¶0140 in the specification. ¶0025 is simply reciting the claim while ¶0140 recites sampling from the first label instead of the third label. Therefore, there is a lack of written description for limitation ii (directed towards sampling from the third label).
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 7-10, the term “largest number” is a relative term which renders the claim indefinite. The term “largest number” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “largest number” could refer to the data set with the most elements, the most frequent element, or the data set with the largest index (i.e. created last). The Examiner will interpret “largest number” to mean the most frequent element.
Regarding claim 10, limitation ii is indefinite. It is unclear whether limitation ii should be sampling from the third label or first label as ¶0025 of the specification (which recites the language of claim 10) indicates the third label but ¶0140 indicates sampling from the first label. The Examiner will interpret the “third label” as a label that matches the most frequent element.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because “a recording medium storing a computer-readable program” as claimed in claim 20 is software per se.
Positive Statement Regarding - 35 USC § 101
The Examiner’s 35 U.S.C. 101 analysis recognizes that the claimed subject matter in claims 1-19 is directed to a practical application of a technical solution. The claimed elements, taken as a whole recite specific steps directed towards implementing a machine learning algorithm for determining product defects. Because the claims recite specific, claimed steps and structural elements that produce a tangible technical result, they are not directed to an abstract idea absent additional inventive concept limitations. Accordingly, the record supports a positive 101 determination for the present claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 6, 11, 14, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Shim et al. 2021 (Shim, Dongsub, et al. "Online Class-Incremental Continual Learning with Adversarial Shapley Value." arXiv preprint arXiv:2009.00093.v3 (2021)) (hereafter, “Shim) in view of Niculescu-Mizil et al. (US 2018/0374569) (hereafter, Niculescu-Mizil), and further in view of Hinton et al. (US 2022/0101624) (hereafter, “Hinton”).
Regarding claim 1, Shim discloses a vision inspection device comprising: a memory including a buffer (Page 3, left column, first paragraph, the memory buffer); and [a processor configured to: i) acquire a plurality of divided images by dividing a captured product image into a plurality of pieces], and a new data set including new [normal product type] data and new [defective type] data (Page 2, §Online Class-Incremental Learning, new classes continually from an online data stream. Examiner considers the data from the online data stream as “new data”) corresponding to [the plurality of divided images]; ii) sample at least one buffer data set among a plurality of buffer data sets stored in the buffer (Page 3, left column, first paragraph, a memory buffer M … minibatch BM of samples selected from the memory buffer; Page 4, right column, paragraph 3, we target two types of samples in M for retrieval. Examiner considers the two types of samples to be the “plurality of buffer data sets”); iii) generate a mini batch by combining the sampled buffer data set with the new data set (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer); and iv) determine whether to store the new data set in the buffer by using a [soft nearest neighbor loss (SNNL)] value of the new data set constituting the mini batch (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples), and a cumulative average [SNNL] value of each of the buffer data sets constituting the mini batch (Page 5, §Memory Update…, samples in M having smaller average KNN-SVs).
However, Shim fails to explicitly disclose a processor configured to: i) acquire a plurality of divided images by dividing a captured product image into a plurality of pieces; normal product type ; defective type; and soft nearest neighbor loss (SNNL).
Niculescu-Mizil teaches a processor (¶0023, at least one processor) configured to: i) acquire a plurality of divided images by dividing a captured product image into a plurality of pieces (¶0036, the image of the item … divide the image in multiple smaller portions); normal product type (¶0036, a representation of the item that has no defects or anomalies); defective type (¶0032, when the anomaly detection and tagging system 200 analyzes an image of an item that is different from the a defectless item, the anomaly detection and tagging system 200 will determine where in the image and on the item the difference is from the defectless item. Examiner considers detecting a defect to indicate the image contained “defective type data”).
Both Shim and Niculescu-Mizil are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers and Niculescu-Mizil is directed towards defect detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the image splitting of Niculescu-Mizil into the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
However, neither Shim nor Niculescu-Mizil, whether considered individually or in combination, fail to explicitly disclose soft nearest neighbor loss (SNNL).
Hinton teaches soft nearest neighbor loss (SNNL) (¶0010, determining the soft nearest neighbor loss).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, and Hinton.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, and Hinton to obtain the invention as specified in claim 1.
Regarding claim 2, in which claim 1 is incorporated, Shim discloses wherein when there is a single buffer data with a cumulative average [SNNL value greater than the SNNL value] of the new data set (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers the input batch to be the new data and the samples in M to be the single buffer data), the [processor] is configured to replace the corresponding buffer data set with the new data set and store the replaced new data set in the buffer (Page 5, §Memory Update…, replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples).
However, Shim fails to explicitly disclose processor and SNNL value greater than the SNNL value.
Niculescu-Mizil teaches processor (¶0023, at least one processor).
Both Shim and Niculescu-Mizil are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers and Niculescu-Mizil is directed towards defect detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the image splitting of Niculescu-Mizil into the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
However, neither Shim nor Niculescu-Mizil, whether considered individually or in combination, fail to explicitly disclose SNNL value greater than the SNNL value.
Hinton teaches SNNL value greater than the SNNL value (¶0089, minimize the soft nearest neighbor loss. Hinton teaches minimizing the SNNL which in view of Shim would mean replacing datasets with ones having lower SNNL).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, and Hinton.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, and Hinton to obtain the invention as specified in claim 2.
Regarding claim 6, Shim in view of Niculescu-Mizil and Hinton discloses the vision inspection device of claim 1
However, neither Shim nor Niculescu-Mizil, whether considered individually or in combination, explicitly disclose wherein the processor is configured to calculate the SNNL value according to Equation 1 below, wherein x denotes the representation vector of the input data, y denotes the class information, b denotes the batch, and T denotes the temperature of a hyperparameter.
Hinton teaches wherein the processor is configured to calculate the SNNL value according to Equation 1 below (Eqn. 2-4; ¶0085 S(⋅,⋅) is a similarity measure (e.g., S(p.sub.i,p.sub.j)=|p.sub.i−p.sub.j|.sup.2); ¶0087, R.sub.i represents the ratio of the intra-class variation and the total variation. Eqn. 4 in Hinton is equivalent to Equation 1 in the instant claim. Eqn. 2 and 3 correspond to the numerator and denominator of the log term in Equation 1, respectively), wherein x denotes the representation vector of the input data (¶0051, A data point refers to an ordered collection of numerical values, e.g., a vector; ¶0085, p.sub.i represents the given data point, p.sub.j represents the data point corresponding to index j. Examiner considers p in Hinton to correspond to x in the instant application), y denotes the class information (¶0085, y.sub.i represents the class of the given data point), b denotes the batch (¶0085, b is the total number of data points (e.g., corresponding to a current batch of data points)), and T denotes the temperature of a hyperparameter (¶0085, T is a temperature parameter).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, and Hinton.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, and Hinton to obtain the invention as specified in claim 6.
Regarding claim 11, in which claim 1 is incorporated, Shim discloses a learning processor configured to train one or more [product] classification models to [determine whether a product is normal from the product] image using updated data stored in the buffer (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer. Then, it simply takes a SGD step with the combined batch , followed by an online update of the memory. Examiner considers the SGD (stochastic gradient descent) step as “training” a model. Since training with SGD takes multiple steps, steps after the first SGD pass use “updated data”).
However, Shim fails to explicitly disclose determine whether a product is normal from the product image.
Niculescu-Mizil teaches determine whether a product is normal from the product image (¶0062, If the original image is found to be defect-free, the reconstructed portions can be used to determine an error with, e.g., a loss function, and backpropagate that error to the hidden layers of the reconstruction learning module 423, as discussed above. Thus, the reconstruction learning module 423 can continuously be trained with defect-free product images while concurrently determining if a product has defects. Examiner considers determining if a product has defects to also imply determining if a product is normal since defect-free is the equivalent).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the normal product detection of Niculescu-Mizil into the SNNL of Hinton and the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, and Hinton.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, and Hinton to obtain the invention as specified in claim 11.
Regarding claim 14, Shim discloses a method for operating a vision inspection device including a buffer (Page 3, left column, first paragraph, the memory buffer), the method comprising the steps of: [acquiring, by a processor, a plurality of divided images by dividing a captured product image into a plurality of pieces]; acquiring, [by the processor], a new data set including new [normal product type] data and new [defective type] data (Page 2, §Online Class-Incremental Learning, new classes continually from an online data stream. Examiner considers the data from the online data stream as “new data”) corresponding to [the plurality of divided images]; sampling, [by the processor], at least one buffer data set among a plurality of buffer data sets stored in the buffer (Page 3, left column, first paragraph, a memory buffer M … minibatch BM of samples selected from the memory buffer; Page 4, right column, paragraph 3, we target two types of samples in M for retrieval. Examiner considers the two types of samples to be the “plurality of buffer data sets”); generating, by the processor, a mini batch by combining the sampled buffer data set with the new data set (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer); and determining, by the processor, whether to store the new data set in the buffer by using a [soft nearest neighbor loss (SNNL)] value of the new data set constituting the mini batch (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples), and a cumulative average [SNNL] value of each of the buffer data sets constituting the mini batch (Page 5, §Memory Update…, samples in M having smaller average KNN-SVs).
However, Shim fails to explicitly disclose acquiring, by a processor, a plurality of divided images by dividing a captured product image into a plurality of pieces; normal product type ; defective type; and soft nearest neighbor loss (SNNL).
Niculescu-Mizil teaches acquiring, by a processor (¶0023, at least one processor), a plurality of divided images by dividing a captured product image into a plurality of pieces (¶0036, the image of the item … divide the image in multiple smaller portions); normal product type (¶0036, a representation of the item that has no defects or anomalies); defective type (¶0032, when the anomaly detection and tagging system 200 analyzes an image of an item that is different from the a defectless item, the anomaly detection and tagging system 200 will determine where in the image and on the item the difference is from the defectless item. Examiner considers detecting a defect to indicate the image contained “defective type data”);
Both Shim and Niculescu-Mizil are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers and Niculescu-Mizil is directed towards defect detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the image splitting of Niculescu-Mizil into the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
However, neither Shim nor Niculescu-Mizil, whether considered individually or in combination, fail to explicitly disclose soft nearest neighbor loss (SNNL).
Hinton teaches soft nearest neighbor loss (SNNL) (¶0010, determining the soft nearest neighbor loss).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, and Hinton.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, and Hinton to obtain the invention as specified in claim 14.
Regarding claim 16, in which claim 14 is incorporated, Shim discloses when there is a single buffer data with a cumulative average [SNNL value greater than the SNNL value] of the new data set (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers the input batch to be the new data and the samples in M to be the single buffer data), replacing, by [the processor], the corresponding buffer data set with new data set; and storing, by [the processor], the replaced new data in the buffer (Page 5, §Memory Update…, replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples).
However, Shim fails to explicitly disclose the processor and SNNL value greater than the SNNL value.
Niculescu-Mizil teaches the processor (¶0023, at least one processor).
Both Shim and Niculescu-Mizil are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers and Niculescu-Mizil is directed towards defect detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the image splitting of Niculescu-Mizil into the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
However, neither Shim nor Niculescu-Mizil, whether considered individually or in combination, fail to explicitly disclose SNNL value greater than the SNNL value.
Hinton teaches SNNL value greater than the SNNL value (¶0089, minimize the soft nearest neighbor loss. Hinton teaches minimizing the SNNL which in view of Shim would mean replacing datasets with ones having lower SNNL).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, and Hinton.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, and Hinton to obtain the invention as specified in claim 16.
Regarding claim 18, in which claim 14 is incorporated, Shim discloses training, by a learning processor, one or more [product] classification models to [determine whether a product is normal from the product] image using updated data stored in the buffer (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer. Then, it simply takes a SGD step with the combined batch , followed by an online update of the memory. Examiner considers the SGD (stochastic gradient descent) step as “training” a model. Since training with SGD takes multiple steps, steps after the first SGD pass use “updated data”).
However, Shim fails to explicitly disclose determine whether a product is normal from the product.
Niculescu-Mizil teaches determine whether a product is normal from the product (¶0062, If the original image is found to be defect-free, the reconstructed portions can be used to determine an error with, e.g., a loss function, and backpropagate that error to the hidden layers of the reconstruction learning module 423, as discussed above. Thus, the reconstruction learning module 423 can continuously be trained with defect-free product images while concurrently determining if a product has defects. Examiner considers determining if a product has defects to also imply determining if a product is normal since defect-free is the equivalent).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the normal product detection of Niculescu-Mizil into the SNNL of Hinton and the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, and Hinton.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, and Hinton to obtain the invention as specified in claim 18.
Claims 3, 4, 5, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Shim et al. 2021 (Shim, Dongsub, et al. "Online Class-Incremental Continual Learning with Adversarial Shapley Value." arXiv preprint arXiv:2009.00093.v3 (2021)) (hereafter, “Shim) in view of Niculescu-Mizil et al. (US 2018/0374569) (hereafter, Niculescu-Mizil) and Hinton et al. (US 2022/0101624) (hereafter, “Hinton”) as applied to claims 1 and 14 above, and further in view of Ciusdel et al. (US 2023/0260106) (hereafter, “Ciusdel”).
Regarding claim 3, in which claim 1 is incorporated, Shim discloses wherein when there is a [plurality of] buffer data with a cumulative average [SNNL value greater than the SNNL] value of the new data set (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers the input batch to be the new data and the samples in M to be the single buffer data), the processor is configured to exchange the buffer data [with the largest SNNL value] with new data set and store the exchanged new data in the buffer (Page 5, §Memory Update…, replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples).
However, Shim fails to explicitly disclose a plurality of buffer data; SNNL value greater than the SNNL; and selecting the largest SNNL value.
Hinton teaches SNNL value greater than the SNNL (¶0089, minimize the soft nearest neighbor loss. Hinton teaches minimizing the SNNL which in view of Shim would mean replacing datasets with ones having lower SNNL).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, disclose a plurality of buffer data and selecting the largest SNNL value.
Ciusdel teaches a plurality of buffer data and selecting the largest SNNL value (¶0082, The datasets with the highest scores of dissimilarity can be annotated by a user and included in the training dataset for training an updated model… where cases with high scores of dissimilarity requiring significant editing. Ciusdel teaches selecting the dataset with the highest value when multiple datasets satisfy a condition. In combination with Shim and Hinton, this would disclose updating the dataset with the highest SNNL when a plurality of sets are greater than the new data set).
Shim, Niculescu-Mizil, Hinton, and Ciusdel are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Ciusdel is directed towards data selection in machine learning models. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Ciusdel into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Ciusdel at ¶0029, Advantageously, embodiments described herein enable input medical data, that is or may be input into the medical analysis network, to be flagged where the medical analysis network is not robust for performing the medical analysis task for the input medical data.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Ciusdel.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Ciusdel to obtain the invention as specified in claim 3.
Regarding claim 4, in which claim 1 is incorporated, Shim discloses wherein when there is a [plurality of] buffer data with a cumulative average [SNNL value greater than the SNNL] value of the new data set (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers the input batch to be the new data and the samples in M to be the single buffer data), the processor is configured to exchange one of the buffer data with new data set through random sampling and store the exchanged new data in the buffer (Page 5, §Memory Update…, replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples) through random sampling (Page 5, §Memory Update…, different variations with random MemoryUpdate).
However, Shim fails to explicitly disclose a plurality of buffer data and SNNL value greater than the SNNL.
Hinton teaches SNNL value greater than the SNNL (¶0089, minimize the soft nearest neighbor loss. Hinton teaches minimizing the SNNL which in view of Shim would mean replacing datasets with ones having lower SNNL).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, disclose a plurality of buffer data and selecting the largest SNNL value.
Ciusdel teaches a plurality of buffer data (¶0082, where cases with high scores of dissimilarity requiring significant editing. Ciusdel teaches selecting a dataset when multiple datasets satisfy a condition).
Shim, Niculescu-Mizil, Hinton, and Ciusdel are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Ciusdel is directed towards data selection in machine learning models. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Ciusdel into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Ciusdel at ¶0029, Advantageously, embodiments described herein enable input medical data, that is or may be input into the medical analysis network, to be flagged where the medical analysis network is not robust for performing the medical analysis task for the input medical data.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Ciusdel.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Ciusdel to obtain the invention as specified in claim 4.
Regarding claim 5, in which claim 1 is incorporated, Shim discloses wherein when there is [no] buffer data set with a cumulative average [SNNL value greater than the SNNL] value of the new data set (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers the input batch to be the new data and the samples in M to be the single buffer data), the processor is configured to delete the new data set (Algorithm 1, lines 2-6; Page 5, §Memory Update…, replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers Bn as the “new data”. Since Bn is looped over in lines 2-6, examiner considers the Bn for each loop to be discarded if the MemoryUpdate condition of line 5 is not met).
However, Shim fails to explicitly disclose the negative condition and SNNL value greater than the SNNL.
Hinton teaches SNNL value greater than the SNNL (¶0089, minimize the soft nearest neighbor loss. Hinton teaches minimizing the SNNL which in view of Shim would mean replacing datasets with ones having lower SNNL).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose the negative condition.
Ciusdel teaches the negative condition (¶0083, Where no editing is required, the results can be shown to the user instantaneously. Where editing is needed, the results are updated. Ciusdel teaches different outcomes when the condition is not met (no editing required)).
Shim, Niculescu-Mizil, Hinton, and Ciusdel are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Ciusdel is directed towards data selection in machine learning models. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Ciusdel into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Ciusdel at ¶0029, Advantageously, embodiments described herein enable input medical data, that is or may be input into the medical analysis network, to be flagged where the medical analysis network is not robust for performing the medical analysis task for the input medical data.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Ciusdel.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Ciusdel to obtain the invention as specified in claim 5.
Regarding claim 17, in which claim 14 is incorporated, Shim discloses when there is [no] buffer data set with a cumulative average [SNNL value greater than the SNNL] value of the new data set (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers the input batch to be the new data and the samples in M to be the single buffer data), deleting, by the processor, the new data set (Algorithm 1, lines 2-6; Page 5, §Memory Update…, replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples. Examiner considers Bn as the “new data”. Since Bn is looped over in lines 2-6, examiner considers the Bn for each loop to be discarded if the MemoryUpdate condition of line 5 is not met).
However, Shim fails to explicitly disclose the negative condition and SNNL value greater than the SNNL.
Hinton teaches SNNL value greater than the SNNL (¶0089, minimize the soft nearest neighbor loss. Hinton teaches minimizing the SNNL which in view of Shim would mean replacing datasets with ones having lower SNNL).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose the negative condition.
Ciusdel teaches the negative condition (¶0083, Where no editing is required, the results can be shown to the user instantaneously. Where editing is needed, the results are updated. Ciusdel teaches different outcomes when the condition is not met (no editing required)).
Shim, Niculescu-Mizil, Hinton, and Ciusdel are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Ciusdel is directed towards data selection in machine learning models. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Ciusdel into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Ciusdel at ¶0029, Advantageously, embodiments described herein enable input medical data, that is or may be input into the medical analysis network, to be flagged where the medical analysis network is not robust for performing the medical analysis task for the input medical data.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Ciusdel.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Ciusdel to obtain the invention as specified in claim 17.
Claims 7, 8, 9, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Shim et al. 2021 (Shim, Dongsub, et al. "Online Class-Incremental Continual Learning with Adversarial Shapley Value." arXiv preprint arXiv:2009.00093.v3 (2021)) (hereafter, “Shim) in view of Niculescu-Mizil et al. (US 2018/0374569) (hereafter, Niculescu-Mizil) and Hinton et al. (US 2022/0101624) (hereafter, “Hinton”) as applied to claims 1 and 14 above, and further in view of Chrysakis et al. (Chrysakis, Aristotelis, and Marie-Francine Moens. "Online continual learning from imbalanced data." International Conference on Machine Learning. PMLR, 2020) (hereafter, “Chrysakis”).
Regarding claim 7, in which claim 1 is incorporated, Shim discloses [wherein when the new data set has a first label that matches the largest number of buffer data sets in the buffer, the processor is configured to] generate the mini batch by sampling the buffer data set (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer) [with the first label].
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose wherein when the new data set has a first label that matches the largest number of buffer data sets in the buffer, the processor is configured to select the first label.
Chrysakis teaches wherein when the new data set has a first label that matches the largest number of buffer data sets in the buffer (Algorithm 1, line 6; Page 2, right column, paragraph 5, we will call a class full if it currently is, or has been in one of the previous time steps, the largest class. Once a class becomes full, it remains so in the future. Examiner considers the not full condition in line 6 of algorithm 1 to include cases where the new data falls under the largest number), the processor is configured to select the first label (Algorithm 1, line 7-8. Lines 7 and 8 describe selecting a class with the largest label).
Shim, Niculescu-Mizil, Hinton, and Chrysakis are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Chrysakis is directed towards memory-based classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Chrysakis into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Chrysakis at Abstract, We demonstrate that CBRS outperforms the state-of the-art memory population algorithms.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Chrysakis.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Chrysakis to obtain the invention as specified in claim 7.
Regarding claim 8, in which claim 1 is incorporated, Shim discloses [wherein when the new data set has a second label that has previously matched the largest number of buffer data sets, the processor is configured to] generate the mini batch by sampling the buffer data set (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer) [with the second label].
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose wherein when the new data set has a second label that has previously matched the largest number of buffer data sets, the processor is configured to select the second label.
Chrysakis teaches wherein when the new data set has a second label that has previously matched the largest number of buffer data sets (Algorithm 1, line 10; Page 2, right column, paragraph 5, we will call a class full if it currently is, or has been in one of the previous time steps, the largest class. The else condition on line 10 selects for when the new data belongs to a “full” class which is a class that was previously largest), the processor is configured to select the second label (Algorithm 1, lines 10-16; Page 2, right column, paragraph 5, we will call a class full if it currently is, or has been in one of the previous time steps, the largest class. Line 15 of algorithm 1 describes selecting a ”full” class which is a class that was previously largest).
Shim, Niculescu-Mizil, Hinton, and Chrysakis are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Chrysakis is directed towards memory-based classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Chrysakis into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Chrysakis at Abstract, We demonstrate that CBRS outperforms the state-of the-art memory population algorithms.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Chrysakis.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Chrysakis to obtain the invention as specified in claim 8.
Regarding claim 9, in which claim 1 is incorporated, Shim discloses [wherein when the new data set does not have a first label that currently matches the largest number of buffer data set and does not have a second label that previously matches the largest number of buffer data set, the processor is configured to] generate the mini batch by sampling the buffer data set (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer) [with the second label].
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose wherein when the new data set does not have a first label that currently matches the largest number of buffer data set and does not have a second label that previously matches the largest number of buffer data set, the processor is configured to select the second label.
Chrysakis teaches wherein when the new data set does not have a first label that currently matches the largest number of buffer data set and does not have a second label that previously matches the largest number of buffer data set, (Algorithm 1, line 6; Page 2, right column, paragraph 5, we will call a class full if it currently is, or has been in one of the previous time steps, the largest class. Once a class becomes full, it remains so in the future. Examiner considers the not full condition in line 6 of algorithm 1 to include cases where the new data also that are not largest since only previously largest are excluded), the processor is configured to select the second label (Algorithm 1, line 7-8. Lines 7 and 8 describe selecting a class with the largest label).
Shim, Niculescu-Mizil, Hinton, and Chrysakis are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Chrysakis is directed towards memory-based classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Chrysakis into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Chrysakis at Abstract, We demonstrate that CBRS outperforms the state-of the-art memory population algorithms.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Chrysakis.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Chrysakis to obtain the invention as specified in claim 9.
Regarding claim 10, in which claim 1 is incorporated, Shim discloses [wherein when the new data set does not have a first label that currently matches the largest number of buffer data set and does not have a second label that previously matches the largest number of buffer data set, the processor is configured to: i) acquire a third label that currently matches the largest number of buffer data sets in the buffer; ii) sample the buffer data set that matches the third label]; and iii) generate the mini batch (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer).
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose wherein when the new data set does not have a first label that currently matches the largest number of buffer data set and does not have a second label that previously matches the largest number of buffer data set, the processor is configured to: i) acquire a third label that currently matches the largest number of buffer data sets in the buffer; ii) sample the buffer data set that matches the third label.
Chrysakis teaches wherein when the new data set does not have a first label that currently matches the largest number of buffer data set and does not have a second label that previously matches the largest number of buffer data set (Algorithm 1, line 6; Page 2, right column, paragraph 5, we will call a class full if it currently is, or has been in one of the previous time steps, the largest class. Once a class becomes full, it remains so in the future. Examiner considers the not full condition in line 6 of algorithm 1 to include cases where the new data also that are not largest since only previously largest are excluded), the processor is configured to: i) acquire a third label that currently matches the largest number of buffer data sets in the buffer (Algorithm 1, line 7, find all instances of the largest class. Examiner considers the class to be a label); ii) sample the buffer data set that matches the third label (Algorithm 1, line 8, select from them an instance at random).
Shim, Niculescu-Mizil, Hinton, and Chrysakis are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Chrysakis is directed towards memory-based classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dataset selection of Chrysakis into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to ensure robustness, as suggested by Chrysakis at Abstract, We demonstrate that CBRS outperforms the state-of the-art memory population algorithms.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Chrysakis.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Chrysakis to obtain the invention as specified in claim 10.
Claims 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Shim et al. 2021 (Shim, Dongsub, et al. "Online Class-Incremental Continual Learning with Adversarial Shapley Value." arXiv preprint arXiv:2009.00093.v3 (2021)) (hereafter, “Shim) in view of Niculescu-Mizil et al. (US 2018/0374569) (hereafter, Niculescu-Mizil) and Hinton et al. (US 2022/0101624) (hereafter, “Hinton”) as applied to claims 1 and 14 above, and further in view of Lu (US 2020/0019893).
Regarding claim 12, Shim in view of Niculescu-Mizil and Hinton discloses the vision inspection device of claim 11.
However, neither Shim nor Hinton, whether considered individually or in combination, explicitly disclose a) share one memory buffer to train a plurality of product classification models through the learning processor; b) acquire determination result values for each of the plurality of product classification models from the input product image; and c) calculate the average of the determination result values and output the final determination result of the product whether the product is normal product or defective product.
Niculescu-Mizil teaches output the final determination result of the product whether the product is normal product or defective product (¶0062, If the original image is found to be defect-free, the reconstructed portions can be used to determine an error with, e.g., a loss function, and backpropagate that error to the hidden layers of the reconstruction learning module 423, as discussed above. Thus, the reconstruction learning module 423 can continuously be trained with defect-free product images while concurrently determining if a product has defects).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the normal product detection of Niculescu-Mizil into the SNNL of Hinton and the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose a) share one memory buffer to train a plurality of product classification models through the learning processor; b) acquire determination result values for each of the plurality of product classification models from the input product image; and c) calculate the average of the determination result values.
Lu teaches a) share one memory buffer to train a plurality of product classification models through the learning processor (¶0089, trained on different parts of the same training set. Examiner considers using the same training set to indicate “sharing one memory buffer”); b) acquire determination result values for each of the plurality of product classification models from the input product image (¶0089, The random forest model operates by constructing a multitude of decision trees at training time and outputting a class that is a mode of classes (classification) or a mean prediction (regression) of individual trees. Examiner considers acquiring the mode or mean to indicate acquiring the results of individual models); and c) calculate the average of the determination result values (¶0089, outputting a class that is … a mean prediction (regression) of individual trees).
Shim, Niculescu-Mizil, Hinton, and Lu are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Lu is directed towards multi-model classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the multi-model classifier of Lu into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce variance, as suggested by Lu at ¶0089, with a goal of reducing a variance.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Lu.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Lu to obtain the invention as specified in claim 12.
Regarding claim 19, Shim in view of Niculescu-Mizil and Hinton discloses the method of claim 18.
However, neither Shim nor Hinton, whether considered individually or in combination, explicitly disclose sharing, by the processor, one memory buffer to train a plurality of product classification models through the learning processor; acquiring, by the processor, determination result values for each of the plurality of product classification models from the input product image; and calculating, by the processor, the average of the determination result values and output the final determination result of the product whether the product is normal product or defective product.
Niculescu-Mizil teaches output the final determination result of the product whether the product is normal product or defective product (¶0062, If the original image is found to be defect-free, the reconstructed portions can be used to determine an error with, e.g., a loss function, and backpropagate that error to the hidden layers of the reconstruction learning module 423, as discussed above. Thus, the reconstruction learning module 423 can continuously be trained with defect-free product images while concurrently determining if a product has defects).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the normal product detection of Niculescu-Mizil into the SNNL of Hinton and the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose sharing, by the processor, one memory buffer to train a plurality of product classification models through the learning processor; acquiring, by the processor, determination result values for each of the plurality of product classification models from the input product image; and calculating, by the processor, the average of the determination result values.
Lu teaches sharing, by the processor, one memory buffer to train a plurality of product classification models through the learning processor (¶0089, trained on different parts of the same training set. Examiner considers using the same training set to indicate “sharing one memory buffer”); b) acquiring, by the processor, determination result values for each of the plurality of product classification models from the input product image (¶0089, The random forest model operates by constructing a multitude of decision trees at training time and outputting a class that is a mode of classes (classification) or a mean prediction (regression) of individual trees. Examiner considers acquiring the mode or mean to indicate acquiring the results of individual models); and calculating, by the processor, the average of the determination result values (¶0089, outputting a class that is … a mean prediction (regression) of individual trees).
Shim, Niculescu-Mizil, Hinton, and Lu are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Lu is directed towards multi-model classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the multi-model classifier of Lu into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce variance, as suggested by Lu at ¶0089, with a goal of reducing a variance.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Lu.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Lu to obtain the invention as specified in claim 19.
Claims 13, 15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shim et al. 2021 (Shim, Dongsub, et al. "Online Class-Incremental Continual Learning with Adversarial Shapley Value." arXiv preprint arXiv:2009.00093.v3 (2021)) (hereafter, “Shim) in view of Niculescu-Mizil et al. (US 2018/0374569) (hereafter, Niculescu-Mizil) and Hinton et al. (US 2022/0101624) (hereafter, “Hinton”) as applied to claims 1 and 14 above, and further in view of Park et al. (KR 20190078692) (hereafter, “Park”).
Regarding claim 13, in which claim 1 is incorporated, Shim discloses wherein the processor is configured to sample at least one buffer data set among the plurality of buffer data sets stored in the buffer (Page 3, left column, first paragraph, a memory buffer M … minibatch BM of samples selected from the memory buffer; Page 4, right column, paragraph 3, we target two types of samples in M for retrieval. Examiner considers the two types of samples to be the “plurality of buffer data sets”) [based on a weight according to a preset reference].
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose based on a weight according to a preset reference.
Park teaches based on a weight according to a preset reference (Page 6, first paragraph, sample mini-batches by taking a weight corresponding to the frequency of the rank of each label from the sampled mini batch group into consideration).
Shim, Niculescu-Mizil, Hinton, and Park are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Park is directed towards data selection for models. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the weights of Park into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to generate similar label distributions, as suggested by Park at Page 5, paragraph 6, the label distribution of the entire data and the label distribution of the mini layout group can be set to be similar to each other.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Park.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Park to obtain the invention as specified in claim 13.
Regarding claim 15, in which claim 14 is incorporated, Shim discloses wherein the step of sampling at least one buffer data set comprises the step of sampling at least one buffer data set among the plurality of buffer data sets stored in the buffer (Page 3, left column, first paragraph, a memory buffer M … minibatch BM of samples selected from the memory buffer; Page 4, right column, paragraph 3, we target two types of samples in M for retrieval. Examiner considers the two types of samples to be the “plurality of buffer data sets”) [based on a weight according to a preset reference].
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose based on a weight according to a preset reference.
Park teaches based on a weight according to a preset reference (Page 6, first paragraph, sample mini-batches by taking a weight corresponding to the frequency of the rank of each label from the sampled mini batch group into consideration).
Shim, Niculescu-Mizil, Hinton, and Park are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Park is directed towards data selection for models. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the weights of Park into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to generate similar label distributions, as suggested by Park at Page 5, paragraph 6, the label distribution of the entire data and the label distribution of the mini layout group can be set to be similar to each other.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Park.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Park to obtain the invention as specified in claim 15.
Regarding claim 20, Shim discloses [a recording medium storing a computer-readable program] for executing a method for operating a vision inspection device, wherein the method comprises the steps of: [acquiring a plurality of divided images by dividing a captured product image into a plurality of pieces]; acquiring a new data set including new [normal product type] data and new [defective type] data (Page 2, §Online Class-Incremental Learning, new classes continually from an online data stream. Examiner considers the data from the online data stream as “new data”) corresponding to [the plurality of divided images]; sampling at least one buffer data set among a plurality of buffer data sets stored in the buffer (Page 3, left column, first paragraph, a memory buffer M … minibatch BM of samples selected from the memory buffer; Page 4, right column, paragraph 3, we target two types of samples in M for retrieval. Examiner considers the two types of samples to be the “plurality of buffer data sets”) [based on a weight according to a preset reference]; generating a mini batch by combining the sampled buffer data set with the new data set (Page 3, left column, first paragraph, it concatenates the incoming minibatch Bn with another minibatch BM of samples selected from the memory buffer); and determining whether to store the new data set in the buffer by using a [soft nearest neighbor loss (SNNL)] value of the new data set constituting the mini batch (Page 5, §Memory Update…, we find that samples with high KNN-SV promote clustering effect in the latent space. Therefore, they are useful to store in the memory … replace samples in M having smaller average KNN-SVs than samples in Bn with the input batch samples), and a cumulative average SNNL value of each of the buffer data sets constituting the mini batch (Page 5, §Memory Update…, samples in M having smaller average KNN-SVs).
However, Shim fails to explicitly disclose a processor configured to: i) acquire a plurality of divided images by dividing a captured product image into a plurality of pieces; normal product type ; defective type; based on a weight according to a preset reference; and soft nearest neighbor loss (SNNL).
Niculescu-Mizil teaches a processor (¶0023, at least one processor) configured to: i) acquire a plurality of divided images by dividing a captured product image into a plurality of pieces (¶0036, the image of the item … divide the image in multiple smaller portions); normal product type (¶0036, a representation of the item that has no defects or anomalies); defective type (¶0032, when the anomaly detection and tagging system 200 analyzes an image of an item that is different from the a defectless item, the anomaly detection and tagging system 200 will determine where in the image and on the item the difference is from the defectless item. Examiner considers detecting a defect to indicate the image contained “defective type data”).
Both Shim and Niculescu-Mizil are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers and Niculescu-Mizil is directed towards defect detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the image splitting of Niculescu-Mizil into the memory based model of Shim. The suggestion/motivation for doing so would have been to take corrective action, as suggested by Niculescu-Mizil at ¶0033, information regarding the tagged anomaly can then be provided to an anomaly correction system 300 to take corrective action.
However, neither Shim nor Niculescu-Mizil, whether considered individually or in combination, fail to explicitly disclose based on a weight according to a preset reference and soft nearest neighbor loss (SNNL).
Hinton teaches soft nearest neighbor loss (SNNL) (¶0010, determining the soft nearest neighbor loss).
Shim, Niculescu-Mizil, and Hinton are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, and Hinton is directed towards machine learning classifiers. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the SNNL of Hinton into the image splitting of Niculescu-Mizil and the memory based model of Shim. The suggestion/motivation for doing so would have been to reduce resources used, as suggested by Hinton at ¶0039, the soft nearest neighbor loss may also enable the classification network to be trained using less training data, over fewer training iterations, or both, thereby reducing consumption of computational resources.
However, none of Shim, Niculescu-Mizil, and Hinton, whether considered individually or in combination, explicitly disclose based on a weight according to a preset reference.
Park teaches based on a weight according to a preset reference (Page 6, first paragraph, sample mini-batches by taking a weight corresponding to the frequency of the rank of each label from the sampled mini batch group into consideration).
Shim, Niculescu-Mizil, Hinton, and Park are analogous to the claimed invention because Shim is directed towards memory-based visual classifiers, Niculescu-Mizil is directed towards defect detection, Hinton is directed towards machine learning classifiers, and Park is directed towards data selection for models. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the weights of Park into the SNNL of Hinton, the image splitting of Niculescu-Mizil, and the memory based model of Shim. The suggestion/motivation for doing so would have been to generate similar label distributions, as suggested by Park at Page 5, paragraph 6, the label distribution of the entire data and the label distribution of the mini layout group can be set to be similar to each other.
This method of improving Shim was within the ordinary ability of one of ordinary skill in the art based on the teachings of Niculescu-Mizil, Hinton, and Park.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Shim with the teachings of Niculescu-Mizil, Hinton, and Park to obtain the invention as specified in claim 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sjögren et al. (US 2023/0215195) discloses dividing images and using SNNL for image classification (¶0116, he images may be divided into fixed subimages covering all or part of the original images; ¶0127, From the soft nearest neighbor v).
Yim et al. (US 2022/0352714) discloses using multiple models (¶0126, the random forest model may generate a plurality of tree models based on a relation between input data and output data).
Ji et al. (US 2021/0174482) discloses dividing an input image and detecting defects in the image (¶0028, machine learning (e.g., neural network) model that has been specifically trained to detect possible defects; ¶0050, the overall area of the image has been divided into 72 image blocks).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMAO DING whose telephone number is (571)272-7237. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAOMAO DING/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights