DETAILED ACTION
This action is responsive to the application filed 4/10/2024. Claims 1-12 are pending in the case. Claims 1 and 7 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. The date used in determinations of prior art is 10/12/2021.
Drawings
The drawings are objected to because in the detailed description of Figure 6, certain elements of the figure are identified by color despite the fact that the drawings are in black and white. Corrections to Figure 6 are required to clearly point out the required elements described in the specification without referring to their original color. Note that the applicant cannot merely remove the word “red” so as to recognize the required lines as “dotted lines” because there are many dotted lines throughout the Figure that could refer to the described dotted lines. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
At the top of the specification there is a section header titled “Description” which should be removed as it is superfluous. The title of the invention should be at the top of the page.
In the title of the invention, the intro “Invention Title:,” is not required and should be removed; The applicant may simply write the title of the invention at the top of the specification
The two sections titled “Technical Field” and “Background Art” should be put into a parent section titled “Background of the Invention” or something of the like
On page 2, the section titled “Disclosure” should be renamed to “Brief Summary of the Invention,” or something of the like, in accordance with standard practice
The section titles “Technical Problem,” “Technical Solution,” and “Advantageous Effects” on pages 2 and 3 may be removed, these sections are all consistent with the contents of a “Brief Summary of the Invention” section
The section titled “Modes of the Invention” should be renamed “Detailed Description of the Invention,” or something of the like, in accordance with standard practice
In paragraph [26], “In addition, the terms “…..unit,” “…..er,” “module,” and “block” described in the specification…” should read “In addition, the terms “unit,” “module,” and “block” described in the specification.” The list element “…..er” is unnecessary and confusing; it suggests that any word ending in “er” should be viewed in the manner described. Additionally, the bullet points before unit are superfluous and should be removed.
In paragraph [81], the lines indicating “red dotted lines” should be amended in accordance with any changes made to Figure 6 given the objection above. The present application is only allowed to include black and white drawings which have been supplied. Given that these drawings are black and white, the reader is unable to discern which dotted lines are the red dotted lines.
Throughout the specification there is inconsistent line spacing. The line spacing throughout the entire specification should remain the same. This spacing should be either 1 ½ or double spaced.
Throughout the specification there should be a paragraph break between each section; a new section should not commence on the very next line following the end of the previous section.
Appropriate correction is required.
Claim Objections
Applicant is advised that should claim 10 be found allowable, claim 11 will be objected to under 37 CFR 1.75 as being a substantial duplicate thereof. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claims 1-12 are objected to because of the following informalities:
All claims should be numbered in Arabic numerals (i.e., 1, 2, 3, 4, 5…) instead of the current format used to number the claims, i.e. the word “claim” and the brackets should not be included in the numbering of the claims.
Each claim should be separated by a paragraph break instead of commencing on the line immediately following the end of the previous claim
In claim 1, line 5, “a normal image training unit configured to set weights…” should read “a normal image training unit configured to train and set weight…” so as to ensure it is apparent that the CNN and FC are “trained” as required by recitation of “the trained convolutional neural network” in dependent claims; setting the weights does not necessarily imply that they are trained
In claim 1, lines 11-12, an indication as to the obtainment of a modified CNN should be added so as to ensure that it is apparent that the trained CNN and FC which are pruned are “the modified convolutional neural network” in accordance with its recitation in claim 6. The same objection goes for step (c) of claim 7 with recitation of “the modified convolutional neural network” in the dependent claim 12. For example, claim 1, line 12, could read: “…on the loss gradient size for each filter to obtain a modified convolutional neural network.”
In claim 2 and 8, “a loss gradient calculation unit configured to input the image damaged by the adversarial attack to calculate…” should read “a loss gradient calculation unit configured to input the image damaged by the adversarial attack into the trained convolutional neural network to calculate…” so as to ensure that the reader can discern what the damaged image is being input into. From claim 1 it is clear it is the trained convolutional neural network, but this should carry down to claim 2 for increased clarity. The same rejection follows for claim 8 despite the alternate phrasing.
In claims 2, 7, and 8, “convolution neural network” should read “convolutional neural network” to fix the typographical errors
In claim 7, lines 18-19, step (a), an indication as to the obtainment of a trained CNN should be added so as to ensure that it is apparent that the CNN and FC are “trained” as required by recitation of a “trained convolutional neural network” in dependent claims. For example, “setting weights of a convolution neural network and a FC neural network through learning about a normal image to obtain a trained convolution neural network.” Alternatively, the claim could read “training and setting weights of a convolution neural network…” so as to indicate the training as with the objection to claim 1 above
Throughout the claims, any reference made to “the convolutional neural network” after a training step has been performed should read “the trained convolution neural network” so as to remain consistent in which version of the CNN is used. For example, claim 1, line 11, should read: “a filter pruning unit configured to prune some filters of the trained convolutional neural network.”
Throughout the claims, any reference made to “the loss gradients” should read “the loss gradients,” not simply “the gradients” so as to prevent misconstruction of the claims. For example, claim 3, line 3, should read: “…by calculating an L2 norm of the loss gradients for each filter.”
Claims 7-12 use an alphabetical list of steps to format the claimed method. Although there is nothing inherently wrong with this format, the examiner objects to it so as to suggest an alternate phrasing. Instead of using the alphabetical list, the examiner suggests simply referring to the operations by their summarized action. For example, in claim 8, instead of saying “The image training method of claim 7, wherein the operation (b) includes…” simply say “The image training method of claim 7, wherein acquiring loss gradient sizes cause by an image damaged by an adversarial attack includes…” This alternate phrasing will make each claim easier to parse by not requiring the reader to go back and see what step each letter operation refers to in claim 7
Claims 2-6 and 8-12 further inherit the objections from the claims upon which they depend
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a normal image training unit configured to…” in claim 1
“an adversarial image gradient acquisition unit configured to…” in claim 1 and referenced in claim 2
“a filter pruning unit configured to…” in claim 1 and referenced in claim 4
“a retraining unit configured to…” in claim 1 and referenced in claim 6
“a loss gradient acquisition unit configured to…” in claim 2
“a gradient size acquisition unit for each filter configured to…” in claim 2 and referenced in claim 3
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification* as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
*The examiner notes that as in paragraph [26] of the specification, the corresponding structure for each of the listed claim limitations is understood to comprise any “hardware or software or combinations thereof” that is able to perform the claimed function/steps of the corresponding limitation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 6, 7-8, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et. al. (US 20190244103 A1) in view of Liu et. al. (Liu, Xue, Weijie Xia, and Zhimiao Fan. "A deep neural network pruning method based on gradient L1-norm." 2020 IEEE 6th International Conference on Computer and Communications (ICCC). IEEE, 2020.)
Regarding claim 1, Wang et. al. teaches an image training device that is robust to an image adversarial attack (Abstract, "Systems, methods, and computer readable media are described to train a compressed neural network with high robustness. The neural network is first adversarially pre-trained with both original data as well as data perturbed by adversarial attacks for some epochs." Note that as in paragraph [0125], "In some embodiments, an original input data 3102 may be data sets representing a picture, and perturbed data 3104 may be one or more data sets representing disturbance or noise specifically configured to perturb the picture represented by input data 3102." The examiner comes to understand that the reference presents an image training device that is robust to adversarial attacks, including image adversarial attacks.), the image training device comprising:
a normal image training unit configured to set weights of a convolutional neural network and a fully connected (FC) neural network through learning about a normal image (Paragraph [0087], "The adversarial pre-training engine 102 is configured to… adversarially pre-train the neural network with both original data as well as data perturbed by adversarial attacks for some epochs." The examiner comes to understand that the adversarial pre-training engine (a normal image training unit) trains a neural network (by setting its weights) through learning about the original data. As seen above in paragraph [0125], "original input data 3102 may be data sets representing a picture" which comprises a normal image that has not been damaged by an adversarial attack. Additionally, see paragraph [0054], which describes how an example proposed neural network can use a CNN, "filter pruning for CNNs is proposed to obtain machine learning architectures that are more compatible with the hardware," and Fig. 2 which depicts an example neural network that is fully connected. The present application claims an architecture that includes a CNN and a FC Neural network which is trained and one could reasonably determine an architecture from the current reference that has a CNN including (i.e. connected to) a FC neural network (for output) as in Fig. 2, which is trained.);
an adversarial image gradient acquisition unit configured to input an image damaged by an adversarial attack into the trained convolutional neural network and acquire a loss gradient… caused by the image damaged by the adversarial attack for… the convolutional neural network (Paragraph [0126], "At step 3106, the system may compute a gradient using a loss function based on the neural network model. In some embodiments, the gradient may be calculated based on one or more error values between the output of the neural network model based on the combined input… In some cases, each of the error values may be calculated for a respective neuron in the neural network, such that each neuron has an error value reflecting its individual contribution to the error." The examiner comes to understand that the combined input data comprising perturbed data (images damaged by an adversarial attack, as seen in paragraph [0125]) can be put into the trained neural network (from above) to acquire a loss gradient caused by the perturbed data for each neuron of the neural network. See above for how the neural network architecture can comprise a CNN. Note that with no explicit definition of the adversarial image gradient acquisition unit, the portion of the system that performs the cited functions is understood to be the adversarial image gradient acquisition unit. Additionally, see Fig. 3D which shows a visual representation of the provided citations.);
a filter pruning unit configured to prune some filters of the convolutional neural network based on the loss… for each filter (Paragraph [0088], "The pruning engine 104 is configured to prune “unimportant” weights or filters through criteria based on their magnitudes or other method[s] (e.g., Taylor approximation of the loss function)." The examiner comes to understand that the pruning engine (filter pruning unit) can prune some filters of the CNN (in the training architecture) based on the magnitude (size) of the weights/filter for each filter or a loss measure for each filter. See above for how the filter pruning is proposed for the CNN of the network architecture.); and
a retraining unit configured to retrain a convolutional neural network and a FC neural network modified through the filter pruning using the normal image (Paragraph [0089], "The re-training engine 106 is configured to then re-train the pruned neural network with both clean and perturbed data for more epochs." See above for explanation of how the neural network architecture could comprise a CNN and a FC neural network. Note that the re-training engine (retraining unit) retrains the pruned neural network (i.e., modified through filter pruning) using the clean data (comprising the normal image). As explained above, the clean data or original data is understood to comprise the normal image.)
Wang et. al. does not distinctly* disclose the limitations to:
acquire a loss gradient size… for each filter of the CNN; and
prune some filters of the convolutional neural network based on the loss gradient size for each filter
(*The examiner notes that Wang et al. teaches the method for acquiring a loss gradient, as cited above, as well as pruning filters based on magnitudes or measures of loss, but does not distinctly combine the concepts to acquire a loss gradient size or prune filters based on the loss gradient size for each filter)
However, Liu et. al. teaches these limitations:
…acquire a loss gradient size… for each filter… (II. Related Work, “we adopt a deep network pruning algorithm (GLNP) based on gradient L1-Norm. The core idea of GLNP algorithm is to judge the importance of filter based on gradient L1-norm.” III. Method, B. Gradient L1-Norm Pruning:
PNG
media_image1.png
214
506
media_image1.png
Greyscale
the examiner comes to understand that the GLNP method acquires a loss gradient for each filter in a layer in Step 1 (See equation 1 for using Loss in the calculation of the gradient). In Step 2, the GLNP method determines the size of the loss for each filter using an L1 norm, which includes determining the size of the loss gradient. It would take no inventive effort to alter the gradient acquisition unit cited above to follow this method and calculate the loss gradient size for each filter of the CNN based on am image damaged by an adversarial attack (instead of merely calculating the loss gradients).); and
…prune some filters of the convolutional neural network based on the loss gradient size for each filter (III. Method, B. Gradient L1-Norm Pruning:
PNG
media_image2.png
118
512
media_image2.png
Greyscale
the examiner comes to understand that the GLNP method prunes some filters of the CNN based on the calculated loss gradient size (L1-Norm) for each filter as above. It would take no inventive effort to alter the filter pruning unit of Wang et al. to prune some filters of the CNN based on the loss gradient size for each filter, replacing the Taylor approximation of the loss with the measure of loss gradient size for the pruning criteria.)
Before the effective filing date of the claimed invention, it would have been obvious to one or ordinary skill in the art to combine the image training device of Wang et. al. (including a normal image training unit, adversarial image gradient acquisition unit, filter pruning unit, retraining unit, and their respective operations) with the method for calculating and using a loss gradient size for each filter of a CNN to prune filters of the CNN from Liu et. al. in order to establish a criteria for filter pruning that does not lead to a model with sparse connection (Liu, III. Method, A. Weight and Gradient Parameter Analysis, “The redundant weight parameters can be eliminated by eliminating the weight as small as possible. However, judging the importance of weights according to their size has no theoretical basis and leads to sparse connection… this paper designs a structured pruning method of filter level based on gradient L1 norm. This method provides a new filter evaluation criterion, and does not lead to sparse connection, Moreover, the pruned filters in this method have no influence on the classification results.”) and maintains high accuracy despite a large reduction in parameters (Liu, V. Conclusions, “This paper proposed the pruning method based on gradient L1 norm, and tested the sensitivity of corresponding pruning layers, which gave the pruning ratio for each layer. It achieves about 71% reduction in paras [parameters] for VGG-16 (on CIFAR-10) and deep ResNet without significant loss in the original accuracy.”)
Regarding claim 2, Wang et. al. as modified by Liu et. al. teaches all of the limitations of the image training device of claim 1 as cited above and Wang et. al. further teaches:
…wherein the adversarial image gradient acquisition unit includes: a loss gradient calculation unit configured to input the image damaged by the adversarial attack to calculate loss gradients generated by backpropagating a loss between a feature value output from the FC neural network and a correct answer label (Paragraph [0126], “At step 3106, the system may compute a gradient using a loss function based on the neural network model. In some embodiments, the gradient may be calculated based on one or more error values between the output of the neural network model based on the combined input (e.g. current network output) and the output of the neural network model based on input 3102 alone (e.g. expected network output).” Paragraph [0017], “…This may require the gradients from the loss function (or a cost function) to be backpropagated to the input space to the neural network.” The examiner comes to understand that these cited portions of the reference provide for inputting a damaged input image into the trained model (See above) to calculate loss gradients through backpropagation. In this method, the gradients backpropagated are determined using a loss function that can compare the output of the neural network (feature value) given the combined input (comprising the perturbed image) with the expected output of the neural network (correct answer label) given the normal/clean image. Note that with no explicit definition of a loss gradient calculation unit, the portion of the reference that performs the cited functions is understood to be the loss gradient calculation unit; this unit could be included in the image gradient acquisition unit determined above. Additionally, note that with no explicit definition for feature value, the examiner comes to interpret that a generated label/class could be a feature value for a categorical feature, see paragraph [0125].)
Wang et. al. does not distinctly disclose:
a gradient size acquisition unit for each filter configured to acquire loss gradients for each filter of the convolution neural network and acquire sizes of the acquired loss gradients for each filter
However, Liu et. al. teaches this limitation:
a gradient size acquisition unit for each filter configured to acquire loss gradients for each filter of the convolution neural network and acquire sizes of the acquired loss gradients for each filter (III. Method, B. Gradient L1-Norm Pruning:
PNG
media_image1.png
214
506
media_image1.png
Greyscale
the examiner comes to interpret that for each filter of the CNN a loss gradient is acquired in step 1, as cited above. Then, in step 2 of the method, the sizes of the acquired loss gradients are determined using an L1-Norm calculation. In combination with Wang et. al., with no explicit definition of a gradient size acquisition unit, the examiner comes to understand that the portion of the method in the reference that performs this function is the gradient size acquisition unit which could exist for each filter (as the method is performed for each filter); this unit could be included in the adversarial image gradient acquisition unit determined above.)
See the rationale for modifying Wang et. al. with Liu et. al. in the rejection for claim 1 above which covers use of calculating loss gradient norms or sizes.
Regarding claim 6, Wang et. al. as modified by Liu et. al. teaches all of the limitations of the image training device of claim 2 as cited above and Wang et. al. further teaches:
…wherein the retraining unit resets weights of the modified convolutional neural network by comparing feature values output through the modified convolutional neural network and the FC neural network with the correct answer label (Paragraph [0126], “At step 3106, the system may compute a gradient using a loss function based on the neural network model. In some embodiments, the gradient may be calculated based on one or more error values between the output of the neural network model based on the combined input (e.g. current network output) and the output of the neural network model based on input 3102 alone (e.g. expected network output).” Paragraph [0129], “The gradient may be back propagated into the input space (e.g., through perturbed data set), distributed back into the network layers within the neural network, and used in the calculation or adjustment of the weights of neurons within the neutral network.” Paragraph [0130], “the system may proceed to iteratively train the neural network model with a perturbed data from the updated perturbed data set in each training cycle.” The examiner comes to interpret that as above, the output of the neural network architecture (feature values) can be compared to the output expected based on the normal/clean image (correct answer label) to determine gradients which are used to adjust (reset, for the next training cycle) the weights of the model. Note that with no explicit definition, the examiner comes interpret that resetting the weights of the model could comprise adjusting them for the next training epoch/cycle. Additionally, note that as determine above, the neural network architecture could comprise a CNN and FC neural network, and thereby updating the weights of the neural network architecture comprises updating the weights of the CNN. Finally, note that the re-training unit is understood to perform these training cycles; as seen in paragraph [0089], the re-training unit performs any training epochs beyond the initial training done by the normal image training unit (adversarial pre-training engine).)
Regarding claim 7, see the rejection for claim 1 above. Note that the primary difference between claim 1 and claim 7 is that claim 1 is directed towards a system performing the method whereas claim 7 is directed towards the method performed. The examiner notes that the difference in wording and phrasing of claim 7 with regard to claim 1 does not change the scope of the limitations; all of the citations used to reject claim 1 still apply to claim 7. Additionally, note that the first citation of Wang et. al. used in the rejection for claim 1 applies to methods in addition to systems, i.e. covering the alternate embodiment as a method.
Regarding claim 8, see the rejection for claim 2 above. Note that the primary difference between claim 2 and claim 8 is that claim 2 is directed towards the system that performs the method, introduced in claim 1, whereas claim 8 is directed towards the method performed, introduced in claim 7. The examiner notes that the difference in wording and phrasing of claim 8 with regard to claim 2 does not change the scope of the limitations; all of the citations used to reject claim 2 still apply to claim 8. Additionally, see the rejection for claim 7 above which addresses the alternate embodiment as a method.
Regarding claim 12, see the rejection for claim 6 above. Note that the primary difference between claim 6 and claim 12 is that claim 6 is directed towards the system that performs the method, introduced in claim 1, whereas claim 12 is directed towards the method performed, introduced in claim 7. The examiner notes that the difference in wording and phrasing of claim 12 with regard to claim 6 does not change the scope of the limitations; all of the citations used to reject claim 6 still apply to claim 12. Additionally, see the rejection for claim 7 above which addresses the alternate embodiment as a method.
Claims 3 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et. al. (US 20190244103 A1) in view of Liu et. al. (Liu, Xue, Weijie Xia, and Zhimiao Fan. "A deep neural network pruning method based on gradient L1-norm." 2020 IEEE 6th International Conference on Computer and Communications (ICCC). IEEE, 2020.), further in view of He et. al. (He, Yang, et al. "Soft filter pruning for accelerating deep convolutional neural networks." arXiv preprint arXiv:1808.06866 (2018).).
Regarding claim 3, Wang et. al. as modified by Liu et. al. teaches all of the limitations of the image training device of claim 2 as cited above and Liu et al. teaches:
…wherein the gradient size acquisition unit for each filter acquires gradient sizes for each filter by calculating an… norm of the gradients for each filter. (See the rejection for claim 2 above. The citations provided by Liu et. al. use an L1-Norm to calculate the gradient size for each filter. As above, these functions performed by the reference can comprise a gradient size acquisition unit for each filter which are a part of the adversarial image gradient acquisition unit.)
Wang et. al. as modified by Liu et. al. does not distinctly disclose:
acquiring… sizes… by calculating an L2 norm…
However, He et. al. teaches this limitation:
acquires… sizes… by calculating an L2 norm… (3.2 Soft Filter Pruning (SFP), “After each epoch, the L2-norm of all filters are computed for each weighted layer and used as the criterion of our filter selection strategy.” The examiner comes to interpret that for each filter a size is acquired for the weights using an L2-norm. In combination with references cited above, the L2-norm could be used to replace the L1-norm for determining the size of the loss gradients. The examiner notes that it would take no inventive effort to supplement one size metric/equation with another.)
Before the effective filing date of the claimed invention, it would have been obvious to one or ordinary skill in the art to combine the image training device of Wang et. al. (including a normal image training unit, adversarial image gradient acquisition unit, filter pruning unit, retraining unit, and their respective operations) as modified by the method for calculating and using a loss gradient size for each filter of a CNN to prune filters of the CNN from Liu et. al. with the calculation of an L2-norm for pruning filters in He et. al. in order improve the performance of the pruned model (He, 4.4 Ablation Study, “The performance of L2-norm criteria is slightly better than that of L1-norm criteria. The result of L2-norm is dominated by the largest element, while the result of L1-norm is also largely affected by other small elements. Therefore, filters with some large weights would be preserved by the L2-norm criteria. So the corresponding discriminative features are kept so the performance of the pruned model is better.”)
Regarding claim 9, see the rejection for claim 3 above. Note that the primary difference between claim 3 and claim 9 is that claim 3 is directed towards the system that performs the method, introduced in claim 1, whereas claim 9 is directed towards the method performed, introduced in claim 7. The examiner notes that the difference in wording and phrasing of claim 9 with regard to claim 3 does not change the scope of the limitations; all of the citations used to reject claim 3 still apply to claim 9. Additionally, see the rejection for claim 7 above which addresses the alternate embodiment as a method.
Claims 4-5 and 10-11 are rejected under 35 U.S.C. 103 are being unpatentable over Wang et. al. (US 20190244103 A1) in view of Liu et. al. (Liu, Xue, Weijie Xia, and Zhimiao Fan. "A deep neural network pruning method based on gradient L1-norm." 2020 IEEE 6th International Conference on Computer and Communications (ICCC). IEEE, 2020.), further in view of Madaan et al. (Madaan, Divyam, Jinwoo Shin, and Sung Ju Hwang. "Adversarial neural pruning with latent vulnerability suppression." International conference on machine learning. PMLR, 2020.) and further in view of Seetharaman et al. (Seetharaman, Prem, et al. "Autoclip: Adaptive gradient clipping for source separation networks." 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2020.)
Regarding claim 4, Wang et. al. as modified by Liu et. al. teaches all of the limitations of the image training device of claim 2 as cited above and Liu et al. further teaches:
…gradient sizes… (See the rejections for claims 1 and 2 above)
However, Wang et al. as modified by Liu et al. does not distinctly disclose:
…wherein the filter pruning unit prunes filters whose… sizes are greater…
However, Madaan et al. teaches this limitation:
…wherein the filter pruning unit prunes filters whose… sizes are greater… (4. Adversarial Neural Pruning with Vulnerability Suppression, “The basic idea of ANP is to achieve robustness while suppressing the distortion, by explicitly pruning out the latent features with high distortion,” and 3. Robustness of Deep Representations, “The vulnerability of a k-th latent-feature for l-th layer can be measured by the distortion in that feature in the presence of an adversary. The vulnerability of a latent feature could then be defined as the expectation of the Manhattan distance between the feature value for a clean example (zlk) and its adversarial example (zlk).” The examiner comes to understand that the above provides for measuring a features distortion by determining its distance between a clean and perturbed output value using a Manhattan distance. This distortion is a measure of the vulnerability of the feature as well as an indication of the size of the loss between the clean and perturbed input. During pruning, the features that are more vulnerable or have greater distortions/loss sizes are pruned. The examiner notes that it would take no inventive effort to apply this methodology to the cited references above, wherein the pruning unit prunes filters whose loss gradient sizes (distances) are greater. Additionally, the examiner notes that filters and latent features are understood to be equivalent given each feature is the output of a filter in the CNN, see the visual representation in Figure 1 in which entire filters/features are pruned.)
Before the effective filing date of the claimed invention, it would have been obvious to one or ordinary skill in the art to combine the image training device of Wang et. al. including a normal image training unit, adversarial image gradient acquisition unit, filter pruning unit, retraining unit, and their respective operations) as modified by the method for calculating and using a loss gradient size for each filter of a CNN to prune filters of the CNN from Liu et. al. with the pruning of filters with larger loss in Madaan et al. in order to minimize the vulnerability of the network while achieving robustness and suppressing distortion. (Madaan, 4. Adversarial Neural Pruning with Vulnerability Suppression, “In order to minimize the vulnerability of the network, we first propose Adversarial Neural Pruning (ANP). The basic idea of ANP is to achieve robustness while suppressing the distortion.”)
Wang et al. as modified by Liu et al. and further modified by Madaan et al. does not distinctly* disclose the limitation to:
prune… gradient sizes… greater than or equal to a threshold
(*The examiner notes that as described by the citations of Madaan et al. above, the filters with greater values are pruned, but the reference does not include exact language for determining a threshold to compare to for pruning those greater values.)
However, Seetharaman et al. teaches this limitation:
PNG
media_image3.png
108
417
media_image3.png
Greyscale
prune… gradient sizes… greater than or equal to a threshold… (2. Gradient Clipping, “Gradient clipping enforces an upper bound on the update of θ, by placing a max on the norm of the gradient:
…Note that this clipping scheme is the so-called clip by norm, not clip by value, where individual values of the gradient vectors are clipped if they go beyond a pre-set value.” The examiner comes to understand that the above provides for preventing exploding gradients from affecting the training of a neural network by removing the effect of those gradients (whose magnitudes are larger than a threshold value) on subsequent training of the neural network. Additionally, the examiner interprets that the goal of pruning filters with gradient sizes above a threshold goes hand and hand with this interpretation, i.e. removing filters with large loss gradients prevents those filters from affecting subsequent use and training of the neural network because those filters negatively affect the goal of making the network adversarially robust. The examiner asserts that it would take no inventive effort to apply the cited threshold technique to the combination of references above in order to remove filters whose gradients sizes go beyond that threshold value instead of merely clipping those gradients.)
Before the effective filing date of the claimed invention, it would have been obvious to one or ordinary skill in the art to combine the image training device of Wang et. al. (including a normal image training unit, adversarial image gradient acquisition unit, filter pruning unit, retraining unit, and their respective operations) as modified by the method for calculating and using a loss gradient size for each filter of a CNN to prune filters of the CNN from Liu et. al. and further modified by the pruning of filters with larger loss in Madaan et al. with the adaptive thresholding technique for gradient norms of Seetharaman et al. in order to maintain optimization of a network by not allowing large loss gradients to affect further network training (Seetharaman, 2. Gradient Clipping, “this places a maximum on the step size that can be taken during training, preventing the optimization from going too far in the direction of a gradient with very large magnitude.”) and allow training to ramp up so as improve optimization, reduce training loss, and increase test performance. (Seetharaman, 5. Results and Discussion, “This [the adaptive threshold] manifests as an initial ramp in the step size, similar to learning rate warmup, another popular trick for deep networks [24]. After this initial stage, the step size then decays slowly with little variance. This corresponds to better optimization, as shown by the lower training loss, and higher test performance (9.2 dB vs 8.1 dB).”)
Regarding claim 5, Wang et al. as modified by Liu et al. and Madaan et al., and further modified by Seetharaman et al. teaches all of the limitations of the image training device of claim 4 as cited above and Liu et al. further teaches:
…the gradient sizes of the filters (See the rejections for claims 1 and 2 above)
While Seetharaman et al. further teaches:
wherein the threshold value is set adaptively based on the gradient sizes (6. Conclusion, “We have presented Autoclip, a simple method for adaptively choosing a threshold for gradient clipping based on the history of gradient norms observed during training.” The examiner comes to interpret that during each round of iterative training, as seen in algorithm 1, the threshold value is set adaptively using the new set of gradient norms (some having been clipped). The examiner asserts that it would take no inventive effort to apply this adaptive methodology to the pruning of filters based on those with loss gradients larger than an adaptive threshold. One could determine a threshold using the loss gradient of the current filters during each training or retaining round and use that to adaptively prune filters for that iteration, changing the threshold determined in the next round by removing gradients that negatively impact that threshold.)
See the rationale for combining Wang et al. as modified by Liu et al. and Madaan et al. with Seetharaman et al. in the rejection for claim 4 above. Note that the cited rationale covers the use of an adaptive threshold.
Regarding claims 10 and 11, see the rejection for claim 4 above. Note that the primary difference between claim 4 and claim 10 is that claim 4 is directed towards the system that performs the method, introduced in claim 1, whereas claim 10 is directed towards the method performed, introduced in claim 7. The examiner notes that the difference in wording and phrasing of claim 10 with regard to claim 4 does not change the scope of the limitations; all of the citations used to reject claim 4 still apply to claim 10. Additionally, see the rejection for claim 7 above which addresses the alternate embodiment as a method. Note that as determined above, claim 11 is a substantial duplicate of claim 10 and therefore follows the same rejection.
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's
disclosure. Wang, Chaoqi et al. discloses a network pruning criteria that relies on the gradient norms of a neural network for determining what to prune. The pruning criteria, GraSP, focuses on maintaining gradient flow before and after pruning a network by using changes in gradient norms to score weights and subsequently removing weights whose scores are the largest, i.e. have the biggest change in gradient norm for initial parameters. This method also makes use of threshold for determining which percent of the largest scoring weights to remove. Azarian et al. teaches a method for using an adaptive threshold for pruning weights of a neural network. The method learns a threshold for each layer of the network using gradient descent to calculate a loss gradient with respect to the current layer being pruned. Sehwag et al. is the most pertinent piece of art in regard to the primary concept of the claimed application. The paper discusses HYDRA which is an approach to network pruning that maintains a robust training objective by formulating a risk minimization problem solved using gradient descent. The pruning task specifically makes use of the goal to minimize the empirical loss over the training data. The method includes steps for adversarially training a network using clean and perturbed images, scoring layers of the network, and pruning connections using a pruning mask that minimizes loss. Additionally, the method uses a threshold for determining a percentile of connections to be pruned. Concluding remarks in the article support use of the methodology to prune filters of a neural network instead of connections. Lubana et al. teaches a network pruning technique that focuses on measuring the relationship between gradient flow and known pruning techniques like magnitude based pruning, loss-preservation based pruning, and gradient-norm based pruning; the latter two being most prevalent to the present application. One key observation of the reference teaches how a determined importance score for network pruning removes parameters that maximally increase model loss. Dhillon et al. teaches a Stochastic Activation Pruning strategy with the goal of making a network adversarially robust.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Zane Rawlings whose telephone number is (571)270-3372. The examiner can normally be reached M-F, 10am-7pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Z.A.R./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights