Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
The Examiner encourages Applicant to schedule an interview to discuss issues related to, for example, the rejections noted below under 35 U.S.C § 101 and § 103, for moving toward allowance.
Providing supporting paragraph(s) for each limitation of amended/new claim(s) in Remarks is strongly requested for clear and definite claim interpretations by Examiner.
Priority
Acknowledgment is made of applicant's claim for the provisional application filed on 10/05/2021.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are:
Claim 8 “at least one processing device configured to: obtain a parameter matrix associated with a linear layer of a first machine learning model and containing parameter values for parameters of the linear layer of the first machine learning model” (Note that pars 35-45 of the present application along with fig 1 describe(s) a sufficient structure for performing the claimed function.)
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 2-5, 9-12, 16-19 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim(s) 2 recite(s) the limitation “the importance value” (2nd last line). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to. It appears it may need to read “an importance value”, or something else. For the purposes of examination, “an importance value” is used. In addition, claim(s) 9, 16 is/are rejected for the same reason.
Claim(s) 2 recite(s) the limitation “the parameter value” (2nd last line). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to. It appears it may need to read “a parameter value”, or something else. For the purposes of examination, “a parameter value” is used. In addition, claim(s) 9, 16 is/are rejected for the same reason.
Claim(s) 3 recite(s) the limitation “the initial importance values” (line 6). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to. It appears it may need to read “initial importance values”, or something else. For the purposes of examination, “initial importance values” is used. In addition, claim(s) 10, 17is/are rejected for the same reason.
Claim(s) 2-3, 9-10, 16-17 each recite(s) limitations that raise issues of indefiniteness as set forth above, and their dependent claims are rejected at least based on their direct and/or indirect dependency from the claims listed above. Appropriate explanation and/or amendment is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 21-24, 26-27 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 21
Step 1: “Is the claim to a process, machine, manufacture, or composition of matter?”
The claim is directed to a method. Therefore, yes.
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?”
performing, …, an action based on the prediction; (i.e., mental process)
The claim is directed to an abstract idea. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
obtaining, using at least one processing device of an electronic device, input data; (insignificant extra-solution activity of receiving data, see MPEP 2106.05(g), and well-understood, routine, and conventional generic computer, see MPEP 2106.05(f))
providing, using the at least one processing device, the input data to a compressed machine learning model in order to generate a prediction; and (insignificant extra-solution activity of mere data gathering, see MPEP 2106.05(g), and well-understood, routine, and conventional generic computer, see MPEP 2106.05(f))
using the at least one processing device (well-understood, routine, and conventional generic computer, see MPEP 2106.05(f))
wherein the compressed machine learning model comprises first and second linear layers; and (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
wherein parameter values of the first and second linear layers are based on factorized matrices corresponding to a parameter matrix containing parameter values of a linear layer of a larger machine learning model, the factorized matrices based on importance values corresponding to the parameter values of the linear layer of the larger machine learning model. (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Specifically, the claimed inventions simply append well-understood, routine and conventional activities previously known to the industry, both when viewed independently and as an ordered combination, specified at a high level of generality, to the judicial exception, (e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry). Therefore, no.
Regarding claim 22
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 21. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
the input data comprises at least one of: text data, audio data, and image data (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception (e.g., based on Field of Use and Technological Environment, see MPEP 2106.05(h)). Therefore, no.
Regarding claim 23
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 21. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
the prediction comprises at least one of: a speech-to-text prediction, a text sentiment, and an image classification (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception (e.g., based on Field of Use and Technological Environment, see MPEP 2106.05(h)). Therefore, no.
Regarding claim 24
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 21. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
the action comprises at least one of displaying the prediction to a user, opening an app on the electronic device, and invoking a function of the app on the electronic device (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception (e.g., based on Field of Use and Technological Environment, see MPEP 2106.05(h)). Therefore, no.
Regarding claim 26
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 21. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
the factorized matrices are based on a rank to be preserved, the rank based on one or more characteristics of the electronic device (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception (e.g., based on Field of Use and Technological Environment, see MPEP 2106.05(h)). Therefore, no.
Regarding claim 27
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?”
… process the input data from at least one camera or at least one microphone of the electronic device; (i.e., mental process)
The claim is directed to an abstract idea. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
the compressed machine learning model is used by the electronic device to (well-understood, routine, and conventional generic computer, see MPEP 2106.05(f))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception (e.g., generic computer elements, see MPEP 2106.05(f)). Therefore, no.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 21-24, 26-27 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang et al. (EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis)
Regarding claim 21
A method comprising:
obtaining, using at least one processing device of an electronic device, input data;
(Wang [sec(s) 1] “over-parameterization leads to high computational cost and memory overhead at test time, making it hard to deploy deep neural networks on a resource-limited device.” [sec(s) 5.1] “For networks with skip connections, NN Slimming can only be applied to specially designed network architectures. Therefore, in addition to ResNet32, we also test on PreResNet-29 (He et al., 2016b), which is in the same family of architectures considered by Liu et al. (2017).” [sec(s) 5.3] “We compare EigenDamage to C-OBD, C-OBS, Kron-OBD and Kron-OBS. The results are summarized in Figure 8 in Appendix E due to the space limit. We notice that Eigen-Damage performs slightly better than other baselines with VGGNet and achieves significantly higher performance on ResNet. Specifically, for the results on CIFAR10 dataset with VGGNet, nearly all the methods achieved similar results due to the simplicity of CIFAR10 and VGGNet. How-ever, the performance gap is a bit more clear as the dataset becoming more challenging, e.g., CIFAR100. On a more sophisticated network, ResNet, the performance improvements of EigenDamage were especially significant on CI-FAR10 or CIFAR100. Furthermore, EigenDamage was especially effective in reducing the number of FLOPs, due to the bottleneck structure.” [sec(s) Abs] “With negligible loss of accuracy, an iterative-pruning version gives a 10x reduction in model size and a 8x reduction in FLOPs on wide ResNet32. Our code is available at here”; e.g., “code” read(s) on “using at least one processing device of an electronic device” since code is run on a computer.)
providing, using the at least one processing device, the input data to a compressed machine learning model in order to generate a prediction; and
(Wang [sec(s) 1] “EigenDamage is loss aware. As a consequence, the user need only choose a single compression ratio parameter, and EigenDamage can automatically determine an appropriate rank for each layer, and thus it is calibrated across layers. Empirically, EigenDamage outperforms strong baselines which do pruning in parameter coordinates, especially in more challenging datasets and networks.” [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications.” [sec(s) 4.3] “EigenDamage preserves the input and output shape, and thus can be applied to any convolutional or fully connected architecture without modification, in contrast with Liu et al. (2017), which requires adaptions for networks with cross layer connections. Furthermore, like all Hessian-based pruning methods, our criterion allows us to set one global com pression ratio for the whole network, making it easy to use. Moreover, the introduced eigen-basis QA can be further compressed by the "doubly factored" Kronecker approximation (Ba et al., 2016), and W’ can be also compressed by depth-wise separable decomposition, as detailed in Sections 4.5 and 4.6.”;)
performing, using the at least one processing device, an action based on the prediction;
(Wang [table(s) 1] “One-pass pruning on CIFAR10 and CIFAR100 with VGG19, ResNet32 and PreResNet29. To be noted, we cannot control the pruned ratio of parameters since we prune the whole filter and different filters are not of the same size. We run each experiment five times, and present the mean and standard variance” [sec(s) 1] “EigenDamage is loss aware. As a consequence, the user need only choose a single compression ratio parameter, and EigenDamage can automatically determine an appropriate rank for each layer, and thus it is calibrated across layers. Empirically, EigenDamage outperforms strong baselines which do pruning in parameter coordinates, especially in more challenging datasets and networks.” [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1).” [sec(s) 4.3] “EigenDamage preserves the input and output shape, and thus can be applied to any convolutional or fully connected architecture without modification, in contrast with Liu et al. (2017), which requires adaptions for networks with cross layer connections. Furthermore, like all Hessian-based pruning methods, our criterion allows us to set one global com pression ratio for the whole network, making it easy to use. Moreover, the introduced eigen-basis QA can be further compressed by the "doubly factored" Kronecker approximation (Ba et al., 2016), and W’ can be also compressed by depth-wise separable decomposition, as detailed in Sections 4.5 and 4.6.”; e.g., “action” read(s) on “classification”.)
wherein the compressed machine learning model comprises first and second linear layers; and
(Wang [fig(s) 1] “QTA”, “W’” and “QS”, “Bottleneck” [fig(s) 3] [algorithm 1] [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “fast computation of inverse and eigen-decomposition:
PNG
media_image1.png
93
565
media_image1.png
Greyscale
(5) where Q and Λ are eigenvectors and eigenvalues. Since QS ⊗ QA gives the eigenbasis of the Kronecker product, we call it the Kronecker-factored Eigenbasis (KFE).” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. … each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.” [sec(s) 4.5] “We note that approximation, QA can be efficiently implemented by 1x1 conv, resulting in compact bottleneck structures like ResNet (He et al., 2016a), as shown in Figure 1. This will greatly reduces the size of eigen-basis to be 1/k4 of the original one.”;)
wherein parameter values of the first and second linear layers are based on factorized matrices corresponding to a parameter matrix containing parameter values of a linear layer of a larger machine learning model, the factorized matrices based on importance values corresponding to the parameter values of the linear layer of the larger machine learning model.
(Wang [fig(s) 1] “QTA”, “W’” and “QS”, “Bottleneck” [fig(s) 3] [algorithm 1] [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “fast computation of inverse and eigen-decomposition:
PNG
media_image1.png
93
565
media_image1.png
Greyscale
(5) where Q and Λ are eigenvectors and eigenvalues. Since QS ⊗ QA gives the eigenbasis of the Kronecker product, we call it the Kronecker-factored Eigenbasis (KFE).” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. … each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.” [sec(s) 4.5] “We note that approximation, QA can be efficiently implemented by 1x1 conv, resulting in compact bottleneck structures like ResNet (He et al., 2016a), as shown in Figure 1. This will greatly reduces the size of eigen-basis to be 1/k4 of the original one.”;)
Regarding claim 22
Wang teaches claim 21.
Wang further teaches
wherein the input data comprises at least one of: text data, audio data, and image data.
(Wang [sec(s) 5.2] “Results on Tiny-ImageNet dataset. Apart from the results on CIFAR datasets, we futher test our methods on a more challenging dataset, Tiny-ImageNet, with VGGNet. Tiny-ImageNet consists of 200 classes and 500 images per class for training, and 10,000 images for testing, which are down-sampled from the original ImageNet dataset. The results are in Table 2. Again, EigenDamage outperforms all the baselines by a significant margin”;)
Regarding claim 23
Wang teaches claim 21.
Wang further teaches
wherein the prediction comprises at least one of: a speech-to-text prediction, a text sentiment, and an image classification.
(Wang [sec(s) 5.2] “Results on Tiny-ImageNet dataset. Apart from the results on CIFAR datasets, we futher test our methods on a more challenging dataset, Tiny-ImageNet, with VGGNet. Tiny-ImageNet consists of 200 classes and 500 images per class for training, and 10,000 images for testing, which are down-sampled from the original ImageNet dataset. The results are in Table 2. Again, EigenDamage outperforms all the baselines by a significant margin”;)
Regarding claim 24
Wang teaches claim 21.
Wang further teaches
wherein the action comprises at least one of:
displaying the prediction to a user, opening an app on the electronic device, and invoking a function of the app on the electronic device.
(Wang [table(s) 1] “One-pass pruning on CIFAR10 and CIFAR100 with VGG19, ResNet32 and PreResNet29. To be noted, we cannot control the pruned ratio of parameters since we prune the whole filter and different filters are not of the same size. We run each experiment five times, and present the mean and standard variance” [table(s) 2] “One pass pruning on Tiny-ImageNet with VGG19. To be noted, the network for NN Slimming is pretrained with L1 loss as required by the method. See Appendix F for the full results.” [fig(s) 4] “The percentage of remaining weights at each conv layer after one-pass pruning with a ratio of 0.5 on Tiny-ImageNet with VGG19. The legend is sorted in descending order of test accuracy.” [fig(s) 5] “Low-rank approximation results on VGG19 on CIFAR100 and Tiny-ImageNet by varying either pruning ratios or ranks” [sec(s) 5.2] “Results on Tiny-ImageNet dataset. Apart from the results on CIFAR datasets, we futher test our methods on a more challenging dataset, Tiny-ImageNet, with VGGNet. Tiny-ImageNet consists of 200 classes and 500 images per class for training, and 10,000 images for testing, which are down-sampled from the original ImageNet dataset. The results are in Table 2. Again, EigenDamage outperforms all the baselines by a significant margin”;)
Regarding claim 26
Wang teaches claim 21.
Wang further teaches
wherein the factorized matrices are based on a rank to be preserved, the rank based on one or more characteristics of the electronic device.
(Wang [sec(s) 1] “Network pruning (LeCun et al., 1990; Hassibi et al., 1993; Han et al., 2015; Dong et al., 2017; Zeng & Urtasun, 2019) has been identified as an effective technique to improve the efficiency of deep networks for applications with limited test-time computation and memory budgets. … EigenDamage is loss aware. As a consequence, the user need only choose a single compression ratio parameter, and EigenDamage can automatically determine an appropriate rank for each layer, and thus it is calibrated across layers. Empirically, EigenDamage outperforms strong baselines which do pruning in parameter coordinates, especially in more challenging datasets and networks.” [sec(s) 5.2] “To summarize, EigenDamage performs the best across almost all the settings, and the improvements become more significant when the pruning ratio is high, e.g. 90%, especially on more complicated networks, e.g. (Pre)ResNet, which demonstrates the effectiveness of pruning in the KFE. Moreover, EigenDamage adopts the bottleneck structure, which preserves the input and output dimension, as illustrated in Figure 1, and thus can be trivially applied to any fully connected or convolution layer without modification.” [sec(s) 5.4] “Since EigenDamage can also be viewed as low-rank approximation, we compared it with a state-of-the-art low rank method, CP-Decomposition (Lebedev et al., 2014), which computes a low-rank decomposition of the filter into a sum of rank-one tensors. We experimented low-rank approximation for VGG19 on CIFAR100 and Tiny-ImageNet. For CP-Decomposition, we tested it under two settings: (1) we varied the ranks from {0.1, . . . , 1.0, 1.25, 1.5, 2.0} times of the original rank at each layer; (2) we varied ranks in {4, 8, . . . , 512} for computing the approximation6”;)
Regarding claim 27
Wang teaches claim 21.
Wang further teaches
wherein the compressed machine learning model is used by the electronic device to process the input data from at least one camera or at least one microphone of the electronic device.
(Wang [table(s) 1] “One-pass pruning on CIFAR10 and CIFAR100 with VGG19, ResNet32 and PreResNet29. To be noted, we cannot control the pruned ratio of parameters since we prune the whole filter and different filters are not of the same size. We run each experiment five times, and present the mean and standard variance” [table(s) 2] “One pass pruning on Tiny-ImageNet with VGG19. To be noted, the network for NN Slimming is pretrained with L1 loss as required by the method. See Appendix F for the full results.” [fig(s) 4] “The percentage of remaining weights at each conv layer after one-pass pruning with a ratio of 0.5 on Tiny-ImageNet with VGG19. The legend is sorted in descending order of test accuracy.” [fig(s) 5] “Low-rank approximation results on VGG19 on CIFAR100 and Tiny-ImageNet by varying either pruning ratios or ranks” [sec(s) 5.2] “Results on Tiny-ImageNet dataset. Apart from the results on CIFAR datasets, we futher test our methods on a more challenging dataset, Tiny-ImageNet, with VGGNet. Tiny-ImageNet consists of 200 classes and 500 images per class for training, and 10,000 images for testing, which are down-sampled from the original ImageNet dataset. The results are in Table 2. Again, EigenDamage outperforms all the baselines by a significant margin”;)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6-11, 13-18, 20, 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis) in view of Chen et al. (GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking)
Regarding claim 1
Wang teaches
A method comprising:
obtaining, using at least one processing device of an electronic device, a parameter matrix associated with a linear layer of a first machine learning model and containing parameter values for parameters of the linear layer of the first machine learning model;
(Wang [sec(s) 1] “over-parameterization leads to high computational cost and memory overhead at test time, making it hard to deploy deep neural networks on a resource-limited device. … Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications.” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages.” [sec(s) Abs] “With negligible loss of accuracy, an iterative-pruning version gives a 10x reduction in model size and a 8x reduction in FLOPs on wide ResNet32. Our code is available at here”; e.g., “code” read(s) on “using at least one processing device of an electronic device” since code is run on a computer.)
determining, using the at least one processing device, importance values corresponding to the parameter values;
(Wang [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications.” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 4.2] “To do this, we would need to store the Fisher matrix for each filter, which is intractable for large convolutional layers. To overcome this problem, we adopt the K-FAC approximation F=S⊗A, and compute the change in weights as well as the importance in the following way:
PNG
media_image2.png
126
978
media_image2.png
Greyscale
” [sec(s) 2] “fast computation of inverse and eigen-decomposition:
PNG
media_image1.png
93
565
media_image1.png
Greyscale
(5) where Q and Λ are eigenvectors and eigenvalues. Since QS ⊗ QA gives the eigenbasis of the Kronecker product, we call it the Kronecker-factored Eigenbasis (KFE).” [sec(s) 4.3] “we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}.)
generating, using the at least one processing device, factorized matrices such that a product of the importance values and the factorized matrices contains approximated parameter values for the parameters of the linear layer of the first machine learning model; and
(Wang [algorithm 1] [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 4.2] “To do this, we would need to store the Fisher matrix for each filter, which is intractable for large convolutional layers. To overcome this problem, we adopt the K-FAC approximation F=S⊗A, and compute the change in weights as well as the importance in the following way:
PNG
media_image2.png
126
978
media_image2.png
Greyscale
” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. It is easy to show that the Fisher matrix for W’ is diagonal if the assumptions of K-FAC are satisfied (George et al., 2018). We then apply C-OBD (or equivalently C-OBS since the Fisher is close to diagonal) on W’ for both input and output channels. This way, each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.”;)
generating, using the at least one processing device, a second machine learning model representing a compressed version of the first machine learning model, the second machine learning model having first and second linear layers containing parameter values based on the importance values and the factorized matrices;
(Wang [fig(s) 1] “QS” and “QA” [algorithm 1] “Remove rth row (or cth column) in W’ and rth (or cth) eigenbasis in QA (or QS) if ΔLr (or ΔLc) <= τ” [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 4.2] “To do this, we would need to store the Fisher matrix for each filter, which is intractable for large convolutional layers. To overcome this problem, we adopt the K-FAC approximation F=S⊗A, and compute the change in weights as well as the importance in the following way:
PNG
media_image2.png
126
978
media_image2.png
Greyscale
” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. It is easy to show that the Fisher matrix for W’ is diagonal if the assumptions of K-FAC are satisfied (George et al., 2018). We then apply C-OBD (or equivalently C-OBS since the Fisher is close to diagonal) on W’ for both input and output channels. This way, each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.”;)
(Note: Hereinafter, if a limitation has bold brackets (i.e. [·]) around claim languages, the bracketed claim languages indicate that they have not been taught yet by the current prior art reference but they will be taught by another prior art reference afterwards.)
wherein the factorized matrices are generated based on [weighted] errors between the parameter values for the parameters of the linear layer of the first machine learning model and the approximated parameter values, and wherein weights associated with the weighted errors are based on the importance values.
(Wang [fig(s) 1] “QS” and “QA” [algorithm 1] [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 3] “OBD and OBS share the same basic pruning pipeline: first training a network to (local) minimum in error at weight θ*, and then pruning a weight that leads to the smallest increase in the training error. The predicted increase in the error for a change in full weight vector Δθ is:
PNG
media_image3.png
243
1220
media_image3.png
Greyscale
(6) Eqn. (6) is a simple second order Taylor expansion around the local mode, which is essentially the Laplace approximation.” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. … vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. … This way, each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.” [sec(s) 4.6] “Therefore, we again ignore the correlation along the spatial dimension of filters, i.e. sharing the basis for each spatial dimension (see Figure 3 (b)). In particular, we solve the following problem:
PNG
media_image4.png
210
1191
media_image4.png
Greyscale
(18)”; e.g., eq (6) read(s) on “errors” since it represents a predicted loss change between an original model and an approximated model.)
However, Wang does not appear to explicitly teach:
wherein the factorized matrices are generated based on [weighted] errors between the parameter values for the parameters of the linear layer of the first machine learning model and the approximated parameter values, and wherein weights associated with the weighted errors are based on the importance values.
(Note: Hereinafter, if a limitation has one or more bold underlines, the one or more underlined claim languages indicate that they are taught by the current prior art reference, while the one or more non-underlined claim languages indicate that they have been taught already by one or more previous art references.)
Chen teaches
wherein the factorized matrices are generated based on weighted errors between the parameter values for the parameters of the linear layer of the first machine learning model and the approximated parameter values, and wherein weights associated with the weighted errors are based on the importance values.
(Chen [sec(s) 3] “Given a word embedding matrix A, a standard way to compress A while preserving the information is to perform low-rank approximation over A. A low-rank approximation can be acquired by using singular value decomposition (SVD), which achieves the best rank-k approximation: A≈USVT, (1) where U ∈ RN×k, V ∈ RD×k where k < min(D, N) is the target rank, and S is a diagonal matrix of singular values. After the rank-k low-rank approximation, the memory footprint for A reduces from O(ND) to O(Nk +Dk)” [sec(s) 3.2] “Weighted low-rank approximation. Firstly, we introduce a weighted low-rank approximation to compress the embedding matrix A. This will be used to replace original SVD and serves as the basic building block of our proposed algorithm. The main idea is to assign a different weight for each word’s approximation and penalize more for the higher frequency words when constructing low-rank approximation. Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Wang with the weighted errors of Chen.
One of ordinary skill in the art would have been motived to combine in order to significantly outperform traditional compression methods such as low-rank approximation and pruning.
(Chen [sec(s) Abs] “The experimental results show our method can significantly outperform traditional compression methods such as low-rank approximation and pruning.”)
Regarding claim 2
The combination of Wang, Chen teaches claim 1.
Chen further teaches
wherein generating the factorized matrices comprises using an objective function of:
PNG
media_image6.png
107
448
media_image6.png
Greyscale
where:
W represents the parameter matrix containing the parameter values;
A and B represent parameter matrices based on the importance values and the factorized matrices:
i and j are matrix indices; and
PNG
media_image7.png
113
124
media_image7.png
Greyscale
represents the importance value associated with the parameter value in position (i, j) within the parameter matrix.
(Chen [sec(s) 3] “Given a word embedding matrix A, a standard way to compress A while preserving the information is to perform low-rank approximation over A. A low-rank approximation can be acquired by using singular value decomposition (SVD), which achieves the best rank-k approximation: A≈USVT, (1) where U ∈ RN×k, V ∈ RD×k where k < min(D, N) is the target rank, and S is a diagonal matrix of singular values. After the rank-k low-rank approximation, the memory footprint for A reduces from O(ND) to O(Nk +Dk)” [sec(s) 3.2] “Weighted low-rank approximation. Firstly, we introduce a weighted low-rank approximation to compress the embedding matrix A. This will be used to replace original SVD and serves as the basic building block of our proposed algorithm. The main idea is to assign a different weight for each word’s approximation and penalize more for the higher frequency words when constructing low-rank approximation. Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal.”;)
The combination of Wang, Chen is combinable with Chen for the same rationale as set forth above with respect to claim 1.
Regarding claim 3
The combination of Wang, Chen teaches claim 2.
Chen further teaches
determining an initial importance value of each parameter of the linear layer of the first machine learning model using a validation dataset to generate an initial importance parameter matrix;
(Chen [sec(s) 4.1] “We evaluate our method (GroupReduce) on two tasks: language modeling (LM) and neural machine translation (NMT). For LM, we evaluate GroupReduce on two datasets: Penn Treebank Bank (PTB) and One-billion-Word Benchmark (OBW). OBW is introduced by [2], and it contains a vocabulary of 793,471 words with the sentences shuffled and the duplicates removed. … All four models use a 2-layer LSTM. Two of them (OBW and NMT) are based on exiting model checkpoints and the other two (based on PTB) are trained from scratch due to the lack of publicly released model checkpoint. We train a 2-layer LSTM-based language model on PTB from scratch with two setups: PTB-Small and PTB-Large. The LSTM hidden state sizes are 200 for PTB-Small and 1500 for PTB-Large, so are their embedding sizes. … We show the performance of GroupReduce with different numbers of clusters under the PTB-Large setting in the supplementary.” [sec(s) 3.2] “Weighted low-rank approximation. Firstly, we introduce a weighted low-rank approximation to compress the embedding matrix A. This will be used to replace original SVD and serves as the basic building block of our proposed algorithm. The main idea is to assign a different weight for each word’s approximation and penalize more for the higher frequency words when constructing low-rank approximation.” [sec(s) 4.2] “Whenever, the validation perplexity does not drop down, we decrease the learning rate to an order smaller. As shown in Table 3, GroupReduce can compress both the input embedding and softmax layer 5-10 times without losing much accuracy. In particular, GroupReduce compress 6.6 times on the language model trained on OBW benchmark, which saves more than 5 GB memory.”;)
aggregating the initial importance values in each row of the initial importance parameter matrix to generate a diagonal importance parameter matrix containing aggregated importance values; and
(Chen [sec(s) 3] “Given a word embedding matrix A, a standard way to compress A while preserving the information is to perform low-rank approximation over A. A low-rank approximation can be acquired by using singular value decomposition (SVD), which achieves the best rank-k approximation: A≈USVT, (1) where U ∈ RN×k, V ∈ RD×k where k < min(D, N) is the target rank, and S is a diagonal matrix of singular values. After the rank-k low-rank approximation, the memory footprint for A reduces from O(ND) to O(Nk +Dk)” [sec(s) 3.2] “Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal. Although it is known that weighted SVD with element-wise weights does not have a closed-form solution [23], in our case elements in the same row of A are associated with the same weights, which leads to a simple solution. Define Q = diag(
q
1
, ...,
q
N
), then the optimization problem of (2) is equivalent to
PNG
media_image8.png
121
960
media_image8.png
Greyscale
. (3) Therefore, assume all the qi are nonzeros, we can solve (2) by conducting low-rank approximation of QA. Assume [
U
,
-
S
-
,
V
-
] = svd(QA), then (U*, V*) = (Q-1
U
-
S
-
,
V
-
) will be a solution of (2). Therefore solving Eq.(2) is easy and the solution can be immediately computed from SVD of QA.”; e.g., “Q” read(s) on “diagonal importance parameter matrix”.)
using the aggregated importance values contained in the diagonal importance parameter matrix as the importance values
PNG
media_image7.png
113
124
media_image7.png
Greyscale
in the objective function to generate the factorized matrices.
(Chen [sec(s) 3] “Given a word embedding matrix A, a standard way to compress A while preserving the information is to perform low-rank approximation over A. A low-rank approximation can be acquired by using singular value decomposition (SVD), which achieves the best rank-k approximation: A≈USVT, (1) where U ∈ RN×k, V ∈ RD×k where k < min(D, N) is the target rank, and S is a diagonal matrix of singular values. After the rank-k low-rank approximation, the memory footprint for A reduces from O(ND) to O(Nk +Dk)” [sec(s) 3.2] “Weighted low-rank approximation. Firstly, we introduce a weighted low-rank approximation to compress the embedding matrix A. … Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal. Although it is known that weighted SVD with element-wise weights does not have a closed-form solution [23], in our case elements in the same row of A are associated with the same weights, which leads to a simple solution. Define Q = diag(
q
1
, ...,
q
N
), then the optimization problem of (2) is equivalent to
PNG
media_image8.png
121
960
media_image8.png
Greyscale
. (3) Therefore, assume all the qi are nonzeros, we can solve (2) by conducting low-rank approximation of QA. Assume [
U
,
-
S
-
,
V
-
] = svd(QA), then (U*, V*) = (Q-1
U
-
S
-
,
V
-
) will be a solution of (2). Therefore solving Eq.(2) is easy and the solution can be immediately computed from SVD of QA.”; e.g., “Q” read(s) on “diagonal importance parameter matrix”.)
The combination of Wang, Chen is combinable with Chen for the same rationale as set forth above with respect to claim 1.
Regarding claim 4
The combination of Wang, Chen, teaches claim 3.
Chen further teaches
the factorized matrices include first, second, and third matrices;
(Chen [sec(s) 3] “A low-rank approximation can be acquired by using singular value decomposition (SVD), which achieves the best rank-k approximation: A≈USVT, (1) where U ∈ RN×k, V ∈ RD×k where k < min(D, N) is the target rank, and S is a diagonal matrix of singular values.” [sec(s) 3.2] “Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal. Although it is known that weighted SVD with element-wise weights does not have a closed-form solution [23], in our case elements in the same row of A are associated with the same weights, which leads to a simple solution. Define Q = diag(
q
1
, ...,
q
N
), then the optimization problem of (2) is equivalent to
PNG
media_image8.png
121
960
media_image8.png
Greyscale
. (3) Therefore, assume all the qi are nonzeros, we can solve (2) by conducting low-rank approximation of QA. Assume [
U
,
-
S
-
,
V
-
] = svd(QA), then (U*, V*) = (Q-1
U
-
S
-
,
V
-
) will be a solution of (2). Therefore solving Eq.(2) is easy and the solution can be immediately computed from SVD of QA.”;)
generating the factorized matrices further comprises:
(Chen [sec(s) 3] “A low-rank approximation can be acquired by using singular value decomposition (SVD), which achieves the best rank-k approximation: A≈USVT, (1) where U ∈ RN×k, V ∈ RD×k where k < min(D, N) is the target rank, and S is a diagonal matrix of singular values.” [sec(s) 3.2] “Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal. Although it is known that weighted SVD with element-wise weights does not have a closed-form solution [23], in our case elements in the same row of A are associated with the same weights, which leads to a simple solution. Define Q = diag(
q
1
, ...,
q
N
), then the optimization problem of (2) is equivalent to
PNG
media_image8.png
121
960
media_image8.png
Greyscale
. (3) Therefore, assume all the qi are nonzeros, we can solve (2) by conducting low-rank approximation of QA. Assume [
U
,
-
S
-
,
V
-
] = svd(QA), then (U*, V*) = (Q-1
U
-
S
-
,
V
-
) will be a solution of (2). Therefore solving Eq.(2) is easy and the solution can be immediately computed from SVD of QA.”;)
selecting a rank r to be preserved based on an end user device to use the second machine learning model;
(Chen [sec(s) Abs] “Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. … In this paper, we propose GroupReduce, a novel compression method for neural language models, based on vocabulary-partition (block) based low-rank matrix approximation and the inherent frequency distribution of tokens (the power-law distribution of words).”;)
truncating the first matrix to preserve r largest singular values along its diagonal; and
truncating each of the second and third matrices to preserve r columns;
(Chen [sec(s) 3.2] “Weighted low-rank approximation. Firstly, we introduce a weighted low-rank approximation to compress the embedding matrix A. This will be used to replace original SVD and serves as the basic building block of our proposed algorithm. The main idea is to assign a different weight for each word’s approximation and penalize more for the higher frequency words when constructing low-rank approximation. Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal. Although it is known that weighted SVD with element-wise weights does not have a closed-form solution [23], in our case elements in the same row of A are associated with the same weights, which leads to a simple solution. Define Q = diag(
q
1
, ...,
q
N
), then the optimization problem of (2) is equivalent to
PNG
media_image8.png
121
960
media_image8.png
Greyscale
. (3) Therefore, assume all the qi are nonzeros, we can solve (2) by conducting low-rank approximation of QA. Assume [
U
,
-
S
-
,
V
-
] = svd(QA), then (U*, V*) = (Q-1
U
-
S
-
,
V
-
) will be a solution of (2). Therefore solving Eq.(2) is easy and the solution can be immediately computed from SVD of QA.”;)
the first and second matrices are used to generate the parameter values of the first linear layer of the second machine learning model; and
(Chen [sec(s) 3.2] “Weighted low-rank approximation. Firstly, we introduce a weighted low-rank approximation to compress the embedding matrix A. This will be used to replace original SVD and serves as the basic building block of our proposed algorithm. The main idea is to assign a different weight for each word’s approximation and penalize more for the higher frequency words when constructing low-rank approximation. Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal. Although it is known that weighted SVD with element-wise weights does not have a closed-form solution [23], in our case elements in the same row of A are associated with the same weights, which leads to a simple solution. Define Q = diag(
q
1
, ...,
q
N
), then the optimization problem of (2) is equivalent to
PNG
media_image8.png
121
960
media_image8.png
Greyscale
. (3) Therefore, assume all the qi are nonzeros, we can solve (2) by conducting low-rank approximation of QA. Assume [
U
,
-
S
-
,
V
-
] = svd(QA), then (U*, V*) = (Q-1
U
-
S
-
,
V
-
) will be a solution of (2). Therefore solving Eq.(2) is easy and the solution can be immediately computed from SVD of QA.”;)
the third matrix is used to generate the parameter values of the second linear layer of the second machine learning model.
(Chen [sec(s) 3.2] “Weighted low-rank approximation. Firstly, we introduce a weighted low-rank approximation to compress the embedding matrix A. This will be used to replace original SVD and serves as the basic building block of our proposed algorithm. The main idea is to assign a different weight for each word’s approximation and penalize more for the higher frequency words when constructing low-rank approximation. Mathematically, for the i-th word’s frequency to be qi, we want to approximate the embedding A by minimizing
PNG
media_image5.png
155
862
media_image5.png
Greyscale
(2) where k is the reduced rank; Aij is i-th word’s j-th feature; U ∈ RN×k, V ∈ RD×k; Ui and Vj are i-th and j-th row of U and V respectively. Note that here we do not require U, V to be orthonormal. Although it is known that weighted SVD with element-wise weights does not have a closed-form solution [23], in our case elements in the same row of A are associated with the same weights, which leads to a simple solution. Define Q = diag(
q
1
, ...,
q
N
), then the optimization problem of (2) is equivalent to
PNG
media_image8.png
121
960
media_image8.png
Greyscale
. (3) Therefore, assume all the qi are nonzeros, we can solve (2) by conducting low-rank approximation of QA. Assume [
U
,
-
S
-
,
V
-
] = svd(QA), then (U*, V*) = (Q-1
U
-
S
-
,
V
-
) will be a solution of (2). Therefore solving Eq.(2) is easy and the solution can be immediately computed from SVD of QA.”;)
The combination of Wang, Chen is combinable with Chen for the same rationale as set forth above with respect to claim 1.
Regarding claim 6
The combination of Wang, Chen teaches claim 1.
Wang further teaches
the first machine learning model comprises multiple linear layers; and
(Wang [sec(s) 5.1] “For networks with skip connections, NN Slimming can only be applied to specially designed network architectures. Therefore, in addition to ResNet32, we also test on PreResNet-29 (He et al., 2016b), which is in the same family of architectures considered by Liu et al. (2017).” [sec(s) 5.3] “We compare EigenDamage to C-OBD, C-OBS, Kron-OBD and Kron-OBS. The results are summarized in Figure 8 in Appendix E due to the space limit. We notice that Eigen-Damage performs slightly better than other baselines with VGGNet and achieves significantly higher performance on ResNet. Specifically, for the results on CIFAR10 dataset with VGGNet, nearly all the methods achieved similar results due to the simplicity of CIFAR10 and VGGNet. How-ever, the performance gap is a bit more clear as the dataset becoming more challenging, e.g., CIFAR100. On a more sophisticated network, ResNet, the performance improvements of EigenDamage were especially significant on CI-FAR10 or CIFAR100. Furthermore, EigenDamage was especially effective in reducing the number of FLOPs, due to the bottleneck structure.” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}.”)
the second machine learning model comprises multiple pairs of linear layers, each pair of linear layers of the second machine learning model corresponding to one of the linear layers of the first machine learning model and generated using an associated set of factorized matrices.
(Wang [fig(s) 1] “QTA”, “W’” and “QS”, “Bottleneck” [fig(s) 3] [algorithm 1] [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “fast computation of inverse and eigen-decomposition:
PNG
media_image1.png
93
565
media_image1.png
Greyscale
(5) where Q and Λ are eigenvectors and eigenvalues. Since QS ⊗ QA gives the eigenbasis of the Kronecker product, we call it the Kronecker-factored Eigenbasis (KFE).” [sec(s) 3] “The predicted increase in the error for a change in full weight vector Δθ is:
PNG
media_image3.png
243
1220
media_image3.png
Greyscale
(6) Eqn. (6) is a simple second order Taylor expansion around the local mode, which is essentially the Laplace approximation.” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. … each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.” [sec(s) 4.5] “We note that approximation, QA can be efficiently implemented by 1x1 conv, resulting in compact bottleneck structures like ResNet (He et al., 2016a), as shown in Figure 1. This will greatly reduces the size of eigen-basis to be 1/k4 of the original one.”;)
Regarding claim 7
Wang teaches claim 1.
Wang further teaches
obtaining an additional parameter matrix associated with a linear layer of a third machine learning model and containing additional parameter values for parameters of the linear layer of the third machine learning model;
(Wang [algorithm 1] “training” and “Finetune the network on D until convergence” [sec(s) 1] “A typical pruning procedure consists of three stages: 1) train a large, over-parameterized model, 2) prune the trained model according to a certain criterion, and 3) fine-tune the pruned model to regain the lost performance. … Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications.” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 3] “OBD and OBS share the same basic pruning pipeline: first training a network to (local) minimum in error at weight θ*, and then pruning a weight that leads to the smallest increase in the training error.” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages.”;)
determining additional importance values corresponding to the additional parameter values;
(Wang [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications.” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 4.2] “To do this, we would need to store the Fisher matrix for each filter, which is intractable for large convolutional layers. To overcome this problem, we adopt the K-FAC approximation F=S⊗A, and compute the change in weights as well as the importance in the following way:
PNG
media_image2.png
126
978
media_image2.png
Greyscale
” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}.”;)
generating additional factorized matrices such that a product of the additional factorized matrices contains additional approximated parameter values for the parameters of the linear layer of the third machine learning model; and
(Wang [algorithm 1] [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 4.2] “To do this, we would need to store the Fisher matrix for each filter, which is intractable for large convolutional layers. To overcome this problem, we adopt the K-FAC approximation F=S⊗A, and compute the change in weights as well as the importance in the following way:
PNG
media_image2.png
126
978
media_image2.png
Greyscale
” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. It is easy to show that the Fisher matrix for W’ is diagonal if the assumptions of K-FAC are satisfied (George et al., 2018). We then apply C-OBD (or equivalently C-OBS since the Fisher is close to diagonal) on W’ for both input and output channels. This way, each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.”;)
generating a fourth machine learning model representing a compressed version of the third machine learning model, the fourth machine learning model having multiple linear layers containing parameter values based on the additional factorized matrices.
(Wang [fig(s) 1] “QS” and “QA” [algorithm 1] “Remove rth row (or cth column) in W’ and rth (or cth) eigenbasis in QA (or QS) if ΔLr (or ΔLc) <= τ” [sec(s) 1] “Instead of sparse weight matrices, pruning in the KFE leads to a low-rank approximation, or bottleneck structure, in each layer (see Figure 1). While most existing structured pruning methods (He et al., 2017; Li et al., 2016; Liu et al., 2017; Luo et al., 2017) require specialized network architectures, EigenDamage can be applied to any fully connected or convolution layers without modifications” [sec(s) 2] “Considering l-th layer in a neural network whose input activations are a ∈ Rn, weight matrix W
∈
Rnxm, and output s ∈ Rm, we have s = WTa. Therefore, the weight gradient is
∇
WL = a(∇sL)T. With this formula, K-FAC decomposes this layer’s Fisher matrix F with an independence assumption.” [sec(s) 4.2] “To do this, we would need to store the Fisher matrix for each filter, which is intractable for large convolutional layers. To overcome this problem, we adopt the K-FAC approximation F=S⊗A, and compute the change in weights as well as the importance in the following way:
PNG
media_image2.png
126
978
media_image2.png
Greyscale
” [sec(s) 4.3] “we introduce a novel network reparameterization by breaking each linear operation into three stages. Intuitively, the role of the first and third stages is to rotate to the KFE. Considering a single layer with weight W with K-FAC Fisher S⊗A (see Section 2), we can decompose the weight matrix W as the following form: vec{W} = (QS⊗QA)vec{W} = vec{QAW’QTS} (15) where vec{W’} =(QS⊗QA)Tvec{W}. It is easy to show that the Fisher matrix for W’ is diagonal if the assumptions of K-FAC are satisfied (George et al., 2018). We then apply C-OBD (or equivalently C-OBS since the Fisher is close to diagonal) on W’ for both input and output channels. This way, each layer has a bottleneck structure which is a low-rank approximation, which we term eigenpruning.”;)
Regarding claim 8
The claim is a system claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 9
The claim is a system claim corresponding to the method claim 2, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 10
The claim is a system claim corresponding to the method claim 3, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 11
The claim is a system claim corresponding to the method claim 4, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 13
The claim is a system claim corresponding to the method claim 6, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 14
The claim is a system claim corresponding to the method claim 7, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 15
The claim is a computer readable medium claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 16
The claim is a computer readable medium claim corresponding to the method claim 2, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 17
The claim is a computer readable medium claim corresponding to the method claim 3, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 18
The claim is a computer readable medium claim corresponding to the method claim 4, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 20
The claim is a computer readable medium claim corresponding to the method claim 6, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 25
The claim is a method claim corresponding to the method claim 1, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim 1.
Claim(s) 5, 12, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis) in view of Chen et al. (GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking) in view of Shazeer et al. (Adafactor: Adaptive Learning Rates with Sublinear Memory Cost)
Regarding claim 5
The combination of Wang, Chen teaches claim 2.
Chen further teaches
using adaptive [moment] estimation optimization to generate first factorized matrices based on the objective function; and
(Chen [sec(s) 4] “We compare GroupReduce with two standard model compression strategies: low-rank approximation and pruning. These two techniques are widely used for language model compression, such as [17, 19, 18] We compress both input embedding and softmax matrices. For the low-rank approximation approach, we perform standard SVD on the embedding and softmax matrices and obtain the low-rank approximation. For pruning, we set the entires whose magnitude is less than a certain threshold to zero. Note that storing the sparse matrix requires to use the Compressed Sparse Row or Compressed Sparse Column format, the memory usage is thus 2 times the number of non-zeros in the matrix after pruning. After approximation, we retrain the rest of parameters by SGD optimizer with initial learning rate 0.1. Whenever, the validation perplexity does not drop down, we decrease the learning rate to an order smaller. As shown in Table 3, GroupReduce can compress both the input embedding and softmax layer 5-10 times without losing much accuracy. In particular, GroupReduce compress 6.6 times on the language model trained on OBW benchmark, which saves more than 5 GB memory”;)
using stochastic gradient descent optimization to generate second factorized matrices based on the first factorized matrices and the objective function, the second factorized matrices used to generate the second machine learning model.
(Chen [sec(s) 4] “We compare GroupReduce with two standard model compression strategies: low-rank approximation and pruning. These two techniques are widely used for language model compression, such as [17, 19, 18] We compress both input embedding and softmax matrices. For the low-rank approximation approach, we perform standard SVD on the embedding and softmax matrices and obtain the low-rank approximation. For pruning, we set the entires whose magnitude is less than a certain threshold to zero. Note that storing the sparse matrix requires to use the Compressed Sparse Row or Compressed Sparse Column format, the memory usage is thus 2 times the number of non-zeros in the matrix after pruning. After approximation, we retrain the rest of parameters by SGD optimizer with initial learning rate 0.1. Whenever, the validation perplexity does not drop down, we decrease the learning rate to an order smaller. As shown in Table 3, GroupReduce can compress both the input embedding and softmax layer 5-10 times without losing much accuracy. In particular, GroupReduce compress 6.6 times on the language model trained on OBW benchmark, which saves more than 5 GB memory”;)
However, the combination of Wang, Chen does not appear to explicitly teach:
using adaptive [moment] estimation optimization to generate first factorized matrices based on the objective function; and
Shazeer teaches
using adaptive moment estimation optimization to generate first factorized matrices based on the objective function; and
(Shazeer [sec(s) 1] “We propose a way to reduce memory usage while retaining the empirical benefits of adaptivity by maintaining a factored representation of the squared gradient accumulator across training steps. Specifically, by tracking moving averages of the row and column sums of the squared gradients for matrix-valued variables, we are able to reconstruct a low-rank approximation of the exponentially smoothed accumulator at each training step that is optimal with respect to the generalized Kullback-Leibler divergence. For an n × m matrix, this reduces the memory requirements from O(nm) to O(n+m). We demonstrate empirically using Adam on a large-scale machine translation task known for its expensive models that our approach achieves comparable performance to that obtained using full accumulators.” [sec(s) 3] “We present a concrete implementation of Adam with factored second moment accumulators in Algorithm 2 for the case where the parameter set x can be viewed as a single matrix X. In the event that the parameter set is most suitably partitioned into multiple matrices (treating vectors and scalars as special cases), the steps can be performed in parallel for each matrix individually. We present the algorithm with β1 fixed at 0 so as to focus our attention on the second moments. First moments can be included as in Adam without modification if desired.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Wang, Chen with the adaptive moment estimation optimization of Shazeer.
One of ordinary skill in the art would have been motived to combine in order to reduce storage requirements without compromising empirical performance.
(Shazeer [sec(s) 3] “we propose a novel approach in which model structure is exploited in order to reduce storage requirements without compromising empirical performance.”)
Regarding claim 12
The claim is a system claim corresponding to the method claim 5, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Regarding claim 19
The claim is a computer readable medium claim corresponding to the method claim 5, and is directed to largely the same subject matter. Thus, it is rejected for the same reasons as given in the rejections of the method claim.
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Carreira-Perpinan et al. (Model compression as constrained optimization, with application to neural nets. Part I: general framework) teaches a “learning-compression” algorithm, which alternates a learning step of the uncompressed model, independent of the compression type, with a compression step of the model parameters, independent of the learning task, based on a low-rank approximation by the SVD.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEHWAN KIM whose telephone number is (571)270-7409. The examiner can normally be reached Mon - Fri 9:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEHWAN KIM/Examiner, Art Unit 2129
4/14/2026
Prosecution Insights