DETAILED ACTION
This action is in response to the original filing on 11/10/2023. Claims 1-12 are pending for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-9, and 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Ding et al. (‘RepVGG: Making VGG-style ConvNets Great Again’, submitted 2021, hereinafter Ding) in view of Vasu et al. (‘An Improved One millisecond Mobile Backbone’, submitted 2022, hereinafter Vasu).
Ding was cited in the IDS, filed 11/10/2023.
Regarding Claim 1, Ding teaches a method for enhancing kernel reparameterization of a non-linear machine learning model, comprising: ([Abstract] We present a simple but powerful architecture of convolutional neural network, which has a VGG-like inference time body composed of nothing but a stack of 3 x 3 convolution and ReLU, while the training-time model has a multi-branch topology.)
providing a predefined machine learning model; ([Introduction, Page 2, First Col.] RepVGG has the following advantages. The model has a VGG-like plain (a.k.a. feed-forward) topology 1 without any branches, which means every layer takes the output of its only preceding layer as input and feeds the output into its only following layer. The model’s body uses only 3 _ 3 conv and ReLU. The concrete architecture (including the specific depth and layer widths) is instantiated with no automatic search [44], manual refinement [28], compound scaling[35], nor other heavy designs.)
expanding a kernel of the predefined machine learning model with a non-linear network for convolution operation of the predefined machine learning model to generate the non-linear machine learning model; ([Introduction, Page 2, First and Second Col.] Since the benefits of multi-branch architecture are all for training and the drawbacks are undesired for inference, we propose to decouple the training-time multibranch and inference-time plain architecture via structural re-parameterization, which means converting the architecture from one to another via transforming its parameters. To be specific, a network structure is coupled with a set of parameters, e.g., a conv layer is represented by a 4th-order kernel tensor. If the parameters of a certain structure can be converted into another set of parameters coupled by another structure, we can equivalently replace the former with the latter, so that the overall network architecture is changed. Specifically, we construct the training-time RepVGG using identity and 1x1 branches, which is inspired by ResNet but in a different way that the branches can be removed by structural re-parameterization (Fig. 2,4). After training, we perform the transformation with simple algebra, as an identity branch can be regarded as a degraded 1x1 conv, and the latter can be further regarded as a degraded 3x3 conv, so that we can construct a single 3x3 kernel with the trained parameters of the original 3x3 kernel, identity and 1x1 branches and batch normalization (BN) [19] layers. Consequently, the transformed model has a stack of 3x3 conv layers, which is saved for test and deployment. (Taking a 3x3 convolutional network and expanding it into a multi-branch block, a parallel 1x1 branch, and and identity branch. After expansion, it is trained and then collapsed back into a single 3x3 kernel. The non-linear network being the multi-branch block with the ReLU))
training the non-linear machine learning model; ([Introduction, Page 2, Second Col.] After training, we perform the transformation with simple algebra, as an identity branch can be regarded as a degraded 1x1 conv, (The model is trained)) and
reparameterizing the non-linear network back to a kernel for convolution operation of the non-linear machine learning model to generate a reparameterized machine learning model; and ([Introduction, Page 2, Second Col.] After training, we perform the transformation with simple algebra, as an identity branch can be regarded as a degraded 1x1 conv, and the latter can be further regarded as a degraded 3x3 conv, so that we can construct a single 3x3 kernel with the trained parameters of the original 3x3 kernel, identity and 1x1 branches and batch normalization (BN) [19] layers. Consequently, the transformed model has a stack of 3x3 conv layers, which is saved for test and deployment. (After training, the branches are collapsed back into a single 3x3 kernel))
Ding fails to disclose: deploying the reparameterized machine learning model to an edge device.
In the same field of endeavor, Vasu teaches:
deploying the reparameterized machine learning model to an edge device. ([Introduction, Page 1-2] Our goal is to improve the latency cost of efficient architectures while improving their accuracy by identifying key bottlenecks that affect on-device latency. To identify these bottlenecks, we deploy neural networks on an iPhone12 by using CoreML [11] and benchmark their latency costs. Optimization is another bottleneck, especially when training smaller neural networks with limited capacity. This can be alleviated by decoupling train-time and inference-time architectures, i.e. using a linearly over-parameterized model at train-time and re-parameterizing the linear structures at inference [12, 13, 14]. We further alleviate optimization bottleneck by dynamically relaxing regularization throughout training to prevent the already small models from being over-regularized. (An edge device is defined as a device at the edge of a network, which in essence constitutes a phone))
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated the deployment of one such machine learning model to an edge device as taught by Vasu into Ding as both references are in the same field of machine learning, and in particular both address the need to improve the efficiency of neural networks. Doing so would represent a clear improvement in the art by enabling the improvement of latency costs while also improving the accuracy of identifying bottlenecks that effect such latency [Vasu Introduction, Page 1].
Regarding Claim 2, the combination of Ding and Vasu teaches the invention as claimed in Claim 1, including: wherein the non-linear network comprises non-linear activation layers, a squeeze and excitation network, a self-attention network, a channel attention network, a split attention network, and/or a feed-forward network. ([Vasu, Section 3.2, Page 5] Architectural blocks that force synchronization like global pooling operations used in Squeeze-Excite block [38] also affect overall run-time due to synchronization costs. To demonstrate the hidden costs like memory access cost and synchronization cost, we ablate over using skip connections and squeeze-excite blocks in a 30 layer convolutional neural network. In Table 1b, we show how each of these choices contribute towards latency. Therefore we adopt an architecture with no branches at inference, which results in smaller memory access cost. In addition, we limit the use of Squeeze-Excite blocks to our biggest variant in order to improve accuracy. (Usage of squeeze-excite blocks))
Regarding Claim 3, the combination of Ding and Vasu teaches the invention as claimed in Claim 1, including: wherein deploying the reparameterized machine learning model to the edge device is deploying the reparameterized machine learning model to the edge device for classification, object detection, segmentation, or image restoration. ([Vasu, Abstract] Furthermore, we show that our model generalizes to multiple tasks – image classification, object detection, and semantic segmentation with significant improvements in latency and accuracy as compared to existing efficient architectures when deployed on a mobile device. (The model is deployed for multiple tasks, including image classification, object detection, and semantic segmentation))
Regarding Claim 5, the combination of Ding and Vasu teaches the invention as claimed in Claim 1, including: wherein expanding the kernel of the predefined machine learning model with the non-linear network for convolution operation of the predefined machine learning model to generate the non-linear machine learning model is expanding a QxQ kernel of the predefined machine learning model with the non-linear network for convolution operation of the predefined machine learning model to generate the non-linear machine learning model where Q is a positive integer. ([Ding Introduction, Page 2, Second Col.] Specifically, we construct the training-time RepVGG using identity and 1x1 branches, which is inspired by ResNet but in a different way that the branches can be removed by structural re-parameterization (Fig. 2,4). After training, we perform the transformation with simple algebra, as an identity branch can be regarded as a degraded 1x1 conv, and the latter can be further regarded as a degraded 3x3 conv, so that we can construct a single 3x3 kernel with the trained parameters of the original 3x3 kernel, identity and 1x1 branches and batch normalization (BN) [19] layers. Consequently, the transformed model has a stack of 3x3 conv layers, which is saved for test and deployment. (Expansion of a QxQ kernel, wherein Q is equal to 3))
Regarding Claim 6, the combination of Ding and Vasu teaches the invention as claimed in Claim 1, including: wherein the edge device is a mobile device. ([Introduction, Page 1-2] Our goal is to improve the latency cost of efficient architectures while improving their accuracy by identifying key bottlenecks that affect on-device latency. To identify these bottlenecks, we deploy neural networks on an iPhone12 by using CoreML [11] and benchmark their latency costs. Optimization is another bottleneck, especially when training smaller neural networks with limited capacity. This can be alleviated by decoupling train-time and inference-time architectures, i.e. using a linearly over-parameterized model at train-time and re-parameterizing the linear structures at inference [12, 13, 14]. We further alleviate optimization bottleneck by dynamically relaxing regularization throughout training to prevent the already small models from being over-regularized.)
Regarding Claims 7-9 and 11-12, claims 7-9 and 11-12 are product of manufacture claims that correspond to the method claims of Claims 1-3 and 5-6 above, and so are rejected for the same reasons.
Claims 4 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Ding in view of Vasu as applied in Claims 1-3, 5-9, and 11-12 above, and in further view of Wang et al. (‘RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization’, submitted 2022, hereinafter Wang).
Regarding Claim 4, the combination of Ding and Vasu does not teach the invention as claimed in Claim 3, including: wherein the image restoration comprises super resolution and noise reduction.
In the same field of endeavor, Wang teaches: wherein the image restoration comprises super resolution and noise reduction. ([Wang, Abstract] This paper explores training efficient VGG-style super-resolution (SR) networks with the structural re-parameterization technique. [Section 1 Introduction, Page 2] In this work, we aim to directly address the BN issue in SR, and propose a simple yet effective re-parameterizable block with the BN training non-linearity. First, we carefully analyze the performance drop and artifacts caused by BN, and attribute this issue to the train-test inconsistency. Such inconsistency happens at the “patch level” instead of “dataset level”, which is contrary to our common sense. Interestingly, our analysis also shows that this BN issue only appears during inference, while the training process is stable and healthy. Intuitively, this inconsistency could be eliminated by training with population statistics. Therefore, we propose to first train SR networks with mini-batch statistics, as normal BN does, and then switch to using population statistics at the later training period. In such a way, we can enjoy the benefits from faster convergence with BN training, while not being affected by BN artifacts [Section 3 Methodology, Page 2] In this section, we describe our proposed RepSR in details, including the neat base network, the RepSR block and the re-parameterization for efficient inference. The analysis of BN artifacts in super-resolution and the proposed strategy for removing such artifacts are in Sec. 4. (Image restoration comprises removal of artifacts through usage of super-resolution))
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have incorporated image restoration using super-resolution and noise reduction as taught by Wang into the combined methods of Ding and Vasu, as all three references are in the same field of machine learning and all three teach increasing the efficiencies of neural networks. The introduction of this form of image restoration into the combined reference would bring about an obvious improvement in the art by enabling an improvement of the training speed of multi-branched architectures, as well as allowing for increased performance [Wang, Introduction, Page 1].
Regarding Claim 10, claim 10 is a product of manufacture claim that corresponds to the method of Claim 4 above, and so is rejected for the same reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicants’ disclosure.
Amelon et al. (US 20230386045 A1) is in the same field of endeavor, and discusses the reparameterization of kernels.
Sather et al. (US 12579430 B1) is in the same field of endeavor, and discusses the adjustment of kernel sizes in filters.
XIE et al. (CN 114998702 A) is in the same field of endeavor, and discusses increasing kernel size up to fill a 7*7 cavity.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUSTIN A CARDOSO whose telephone number is (571)272-8512. The examiner can normally be reached M-F 7:30 - 5:00, alternate Friday's off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUSTIN CARDOSO/
Patent Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights