DETAILED ACTION
This action is in response to the original filing on November 3, 2023. Claims 1-19 are pending and have been considered below. Claims 1 and 10 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 6 and 15-16 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
The claimed subject matter not described in the specification is:
an intermediate channel between the prefix layer and the parallel operation layer is larger than an input channel to the prefix layer in claims 6 and 15; applicant has not pointed out where the claim is supported, nor does there appear to be a written description of the claim limitation in the application as filed.
an intermediate channel between the postfix layer and the parallel operation layer is larger than an output channel from the postfix layer in claims 6 and 16; applicant has not pointed out where the claim is supported, nor does there appear to be a written description of the claim limitation in the application as filed.
The examiner finds that the specification lacks support for claims 6, 15, and 16. As such, claims 6, 15, and 16 are rejected under 35 U.S.C. 112(a).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1 – The claim is directed to a method: A reparameterization method for initializing a machine learning model, comprising…
Step 2A, Prong 1 – A judicial exception is recited in this claim as it recites mathematical concepts (see MPEP 2106.04(a)(2)(I)):
initializing a prefix layer of a first low dimensional layer in the machine learning model and a postfix layer of the first low dimensional layer… to initialize prefix and postfix layers is to draw random values for the layers’ internal parameters, which is a mathematical concept.
inverting the prefix layer to generate an inverse prefix layer of the first low dimensional layer… to perform “inverting” is to reverse the mathematical calculations of “the prefix layer,” which is a mathematical concept.
inverting the postfix layer to generate an inverse postfix layer of the first low dimensional layer… to perform “inverting” is to reverse the mathematical calculations of “the postfix layer,” which is a mathematical concept.
combining the inverse prefix layer, the first low dimensional layer and the inverse postfix layer to form a high dimensional layer… to perform “combining” is to perform concatenation, which is a mathematical concept.
generating parallel operation layers from the high dimensional layer… to generate layers is to perform decomposition on “the high dimensional layer,” which is a mathematical concept.
assigning initial weights to the parallel operation layers… to assign initial weights is to draw random values for the weights, which is a mathematical concept.
Step 2A, Prong 2 – The claim does not recite any additional elements that integrate the abstract idea into a practical application.
Step 2B – The claim does not recite any additional elements that amount to significantly more than the abstract idea.
Claims 2-9 recite limitations which further narrow the abstract idea of claim 1 by specifying more details of the mathematical concepts that occur:
Regarding claim 2, specifying wherein the machine learning model contains a sequential structure in this manner does not overcome the rejection of claim 1 as modifying “the machine learning model” does not make the abstract idea of claim 1 to not be math.
Regarding claim 3, specifying wherein the high dimensional layer is a sum of the parallel operation layers in this manner does not overcome the rejection of claim 1 as modifying “the high dimensional layer” does not make the abstract idea of claim 1 to not be math.
Regarding claim 4, specifying wherein each of the parallel operation layers is a skip-connection layer, or an MxN convolution layer in this manner does not overcome the rejection of claim 1 as modifying “each of the parallel operation layers” does not make the abstract idea of claim 1 to not be math.
Regarding claim 5, specifying wherein at least one of the parallel operation layers contains a second low dimensional layer in this manner does not overcome the rejection of claim 1 as modifying “one of the parallel operation layers” does not make the abstract idea of claim 1 to not be math.
Regarding claim 6, specifying wherein: at least one of the parallel operation layers is learnable; an intermediate channel between the prefix layer and the parallel operation layer is larger than an input channel to the prefix layer; and an intermediate channel between the postfix layer and the parallel operation layer is larger than an output channel from the postfix layer in this manner does not overcome the rejection of claim 1 as modifying “one of the parallel operation layers,” “an intermediate channel between the prefix layer and the parallel operation layer,” “an input channel to the prefix layer,” “an intermediate channel between the postfix layer and the parallel operation layer,” and “an output channel from the postfix layer” does not make the abstract idea of claim 1 to not be math.
Regarding claim 7, specifying wherein the parallel operation layers are of a same size in this manner does not overcome the rejection of claim 1 as modifying “the parallel operation layers” does not make the abstract idea of claim 1 to not be math.
Regarding claim 8, this claim further narrows the abstract idea of claim 1 to be based on a mathematical concept: wherein assigning the initial weights to the parallel operation layers is performed according to an arbitrary probability distribution… assigning weights to layers according to an arbitrary probability distribution is a mathematical concept.
Regarding claim 9, specifying wherein the low-dimensional layer is a convolution layer, an elementwise operation layer, or a scaling layer, and the high-dimensional layer is a convolution layer, an elementwise operation layer, or a scaling layer in this manner does not overcome the rejection of claim 1 as modifying “the low-dimensional layer” and “the high-dimensional layer” does not make the abstract idea of claim 1 to not be math.
Claims 10-19 are method claims that recite similar limitations to those of the method of claims 1-9, respectively. Therefore, claims 10-19 are rejected under substantially the same rationale as set forth above with respect to claims 1-9, respectively.
Claim Rejections – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (“ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks,” 2021, hereinafter Guo) in view of Kingman et al. (“Glow: Generative Flow with Invertible 1x1 Convolutions,” 2018, hereinafter Kingman) and further in view of in view of Hu et al. (“Online Convolutional Re-parameterization,” 2022, hereinafter Hu).
Regarding claim 1:
Guo teaches a reparameterization method for initializing a machine learning model (Abstract: “We introduce an approach to training a given compact network… we leverage over-parameterization, which typically improves both neural network optimization and generalization”), comprising: initializing a prefix layer of a first low dimensional layer in the machine learning model and a postfix layer of the first low dimensional layer (Page 3, Section 3.1, ¶1 “We propose to linearly expand a convolutional layer by replacing it with a series of convolutional layers,” Page 4, ¶1 “we therefore propose to expand a k × k convolutional layer into 3 consecutive convolutional layers: a 1×1 convolution; a k×k one; and another 1×1 one,” Page 2, Fig. 1 depicts a prefix layer or “a 1×1 convolution,” a first low dimensional layer in the machine learning model or “a k×k” layer, and a postfix layer or “another 1×1” layer).
Regarding the limitation inverting the prefix layer to generate an inverse prefix layer of the first low dimensional layer, Guo teaches the prefix layer (Page 2, Fig. 1 and Page 4, ¶1 “a 1×1 convolution”) and the first low dimensional layer (Page 2, Fig. 1 and Page 4, ¶1 “a k×k one”). However, Guo fails to teach inverting the prefix layer to generate an inverse prefix layer of the first low dimensional layer.
Kingman, in the same field of endeavor, teaches inverting a 1×1 convolutional layer to generate an inverse prefix layer of an affine coupling layer (Page 4, Table 1: “Invertible 1×1 convolution. W : [c×c]. See Section 3.2” row and “Reverse Function” column: “∀i,j : xi,j = W−1yi,j,” Caption: “The three main components of our proposed flow, their reverses… x signifies the input of the layer, and y signifies its output. Both x and y are tensors of shape [h × w × c] with spatial dimensions (h, w) and channel dimension c. With (i, j) we denote spatial indices into tensors x and y,” Page 5, Section 3.2, ¶1 “We propose… a (learned) invertible 1 ×1 convolution, where the weight matrix is initialized as a random rotation matrix,” ¶2 “an invertible 1 × 1 convolution of a h × w × c tensor h with c × c weight matrix W… We initialize the weights W as a random rotation matrix,” Page 4, Fig. 2(a), Caption: “We propose a generative flow… an invertible 1×1 convolution, followed by an affine transformation,” Fig. 2(a) depicts an “invertible 1×1 conv” layer or an inverse prefix layer prior to an “affine coupling layer”).
Regarding the limitation inverting the postfix layer to generate an inverse postfix layer of the first low dimensional layer, Guo teaches the postfix layer (Page 2, Fig. 1 and Page 4, ¶1 “another 1×1 one”) and the first low dimensional layer (Page 2, Fig. 1 and Page 4, ¶1 “a k×k one”). However, Guo fails to teach inverting the postfix layer to generate an inverse postfix layer of the first low dimensional layer.
Kingman, in the same field of endeavor, teaches inverting a 1×1 convolutional layer to generate an inverse postfix layer of an actnorm layer (Page 4, Table 1: “Invertible 1×1 convolution. W : [c×c]. See Section 3.2” row and “Reverse Function” column: “∀i,j : xi,j = W−1yi,j” and Caption, Page 5, Section 3.2, ¶1-2 “We propose… a (learned) invertible 1 ×1 convolution, where… We initialize the weights W as a random rotation matrix,” Page 4, Fig. 2(a), Caption: “We propose a generative flow where each step (left) consists of an actnorm step, followed by an invertible 1×1 convolution,” Fig. 2(a) depicts an “invertible 1×1 conv” layer or an inverse prefix layer prior to an “affine coupling layer”).
Regarding the limitation combining the inverse prefix layer, the first low dimensional layer and the inverse postfix layer to form a high dimensional layer, Guo teaches combining the… prefix layer, the first low dimensional layer and the… postfix layer to form a high dimensional layer (Page 4, ¶2 “the matrix representation of the original layer can be recovered… which encodes a convolution tensor,” Page 2, Fig. 1, Caption: “We propose 3 strategies to linearly expand a compact network. An expanded network can then be contracted back to the compact one algebraically, and outperforms training the compact one,” Fig. 1 depicts the how a prefix layer or “a 1×1 convolution,” the first low dimensional layer or “a k×k” layer, and an inverse postfix layer or “another 1×1” layer can be contracted back into a high dimensional layer or the “original” ConvLayer). However, Guo fails to teach the inverse prefix layer and the inverse postfix layer…
Kingman teaches an inverse prefix layer and an inverse postfix layer (Page 4, Fig. 2(a), Caption: “We propose a generative flow where each step (left) consists of an actnorm step, followed by an invertible 1×1 convolution, followed by an affine transformation,” Table 1: “Invertible 1×1 convolution. W : [c×c]. See Section 3.2” row and “Reverse Function” column: “∀i,j : xi,j = W−1yi,j” and Caption, as explained above; one of ordinary skill in the art may apply the “Reverse Function” used for the “invertible 1×1 convolution” of Kingman to the “1×1 convolution” prefix and “another 1×1” postfix layers of Guo to generate the inverse prefix layer and the inverse postfix layer, respectively).
Guo and Kingman are analogous art to the claimed invention as both are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the invertible 1×1 convolution layer of Kingman with the prefix and postfix 1×1 convolution layers of Guo. The motivation to do so is to improve efficiency and generalization of machine learning models (Kingman, Pages 1-2, Section 1, ¶1 “Two major unsolved problems in the field of machine learning are (1) data-efficiency… and (2) generalization… A promise of generative models… is to overcome these limitations… we work towards this ultimate vision… by aiming to improve upon the state-of-the-art generative models”).
Regarding the limitation generating parallel operation layers from the high dimensional layer, Guo teaches the high dimensional layer (Page 2, Fig. 1 – “ConvLayer,” Page 4, ¶2 “the original layer”). However, the combination of Guo and Kingman fails to teach generating parallel operation layers from the high dimensional layer.
Hu, in the same field of endeavor, teaches generating parallel operation layers from a block (Page 4, Col. 2, Fig. 4 – (b) and ¶2 “no matter how complicated a linear re-param block is, the following two properties always hold… The block can be represented by a series of parallel branches, each of which consists of a sequence of convolutional layers”).
Regarding the limitation assigning initial weights to the parallel operation layers, Kingman teaches assigning initial weights to convolutional layers (Page 5, Section 3.3, ¶2 “We initialize the last convolution of each NN() with zeros,” Page 6, Section 5, ¶1 “we let each NN() have three convolutional layers”). However, the combination of Guo and Kingman fails to teach …the parallel operation layers.
Hu teaches the parallel operation layers (Page 4, Col. 2, ¶2 “a series of parallel branches, each of which consists of a sequence of convolutional layers”).
Guo, Kingman, and Hu are analogous art to the claimed invention as all are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the invertible 1×1 convolution layer and weight initialization of Kingman and the prefix and postfix 1×1 convolution layers and the high dimensional layer of Guo with the generating of parallel operation layers of Hu. The motivation to do so is to improve efficiency and generalization of machine learning models (Kingman, Pages 1-2, Section 1, ¶1 “Two major unsolved problems in the field of machine learning are (1) data-efficiency… and (2) generalization… A promise of generative models… is to overcome these limitations… we work towards this ultimate vision… by aiming to improve upon the state-of-the-art generative models”) and “to save the training-time memory cost by about 70% and accelerate the training speed by around 2×” (Hu, Abstract).
Regarding claim 2, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Guo teaches wherein the machine learning model contains a sequential structure (Page 5, Section 4.1, ¶1 “we show that our method can improve the results of the compact MobileNet… MobileNetV2… and ShuffleNetV2,” wherein the models described are all known CNN architectures in the art, which encompass wherein the machine learning model contains a sequential structure).
Regarding claim 3, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Regarding the limitation wherein the high dimensional layer is a sum of the parallel operation layers, Guo teaches the high dimensional layer (Page 2, Fig. 1 – “ConvLayer,” Page 4, ¶2 “the original layer”). However, the combination of Guo and Kingman fails to teach wherein the high dimensional layer is a sum of the parallel operation layers.
Hu teaches wherein a block is a sum of the parallel operation layers (Page 4, Col. 2, ¶2 “no matter how complicated a linear re-param block is, the following two properties always hold… The block can be represented by a series of parallel branches, each of which consists of a sequence of convolutional layers,” Fig. 4 depicts wherein the output of a block is a sum of the parallel operation layers or the “series of parallel branches” with “a sequence of convolutional layers”).
Guo and Hu are analogous art to the claimed invention as both are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the generating of parallel operation layers of Hu with the high dimensional layer of Guo. The motivation to do so is “to save the training-time memory cost by about 70% and accelerate the training speed by around 2×” (Hu, Abstract).
Regarding claim 4, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Hu teaches wherein each of the parallel operation layers is a skip-connection layer, or an MxN convolution layer (Page 4, Col. 2, ¶2 “a series of parallel branches, each of which consists of a sequence of convolutional layers” and Fig. 4 – (b) depicts parallel MxN convolution layers).
Guo and Hu are analogous art to the claimed invention as both are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the generating of parallel operation layers of Hu with the high dimensional layer of Guo. The motivation to do so is “to save the training-time memory cost by about 70% and accelerate the training speed by around 2×” (Hu, Abstract).
Regarding claim 5, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
The combination of Guo and Hu teach wherein at least one of the parallel operation layers contains a second low dimensional layer (Guo: Page 2, ¶2 “replacing a k×k convolution with k > 3,” Fig. 1 depicts the original “k×k convolution” ConvLayer or the high dimensional layer; Hu: Page 4, Col. 2, ¶2 “a series of parallel branches, each of which consists of a sequence of convolutional layers” and Fig. 4 – (b) depicts convolutional layers of dimensions 1 × 1 to 5 × 5, or layers having lower dimensions than the original k×k convolution or high dimensional layer of Guo, hence at least one of the parallel operation layers of Hu contains a second low dimensional layer, when given its broadest reasonable interpretation).
Guo and Hu are analogous art to the claimed invention as both are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the generating of parallel operation layers of Hu with the high dimensional layer of Guo. The motivation to do so is “to save the training-time memory cost by about 70% and accelerate the training speed by around 2×” (Hu, Abstract).
Regarding claim 6, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
The combination of Guo and Kingman fails to teach wherein: at least one of the parallel operation layers is learnable. However, Hu teaches this limitation (Page 4, Col. 1, Section 3.3, ¶1 “we describe the standard procedure for squeezing a training-time linear block into a single convolutional kernel,” Col. 2, Fig. 4 – (b) and ¶2 “no matter how complicated a linear re-param block is, the following two properties always hold… The block can be represented by a series of parallel branches, each of which consists of a sequence of convolutional layers,” ¶3 “With the above two properties, we can squeeze a block if we can simplify… a multi-branch (i.e., the parallel structure),” wherein a “training-time linear block” with “a series of parallel branches” implies at least one of the parallel operation layers is learnable).
Regarding the limitation an intermediate channel between the prefix layer and the parallel operation layer is larger than an input channel to the prefix layer, Guo teaches an intermediate channel between the prefix layer and the low dimensional layer is larger than an input channel to the prefix layer (Page 4, ¶1 “we therefore propose to expand a k × k convolutional layer into 3 consecutive convolutional layers: a 1×1 convolution; a k×k one; and another 1×1 one… this allows us to increase… the number of channels by setting p,q > n,m… for an original layer with m input channels and n output ones, given an expansion rate r, we define the number of output channels of the first 1 × 1 layer as p = rm,” Page 5, Section 4.1.1, ¶1 “In this set of experiments, the expansion rate r is set to 4,” wherein the number of input channels, or “m,” is less than the number of output channels, or “p = rm,” of the first 1×1 layer, hence an intermediate channel between the prefix layer and the low dimensional layer, or “p,” is larger than an input channel to the prefix layer, or “m”). However, the combination of Guo and Kingman fails to teach …the parallel operation layer…
Hu teaches the parallel operation layer (Page 4, Col. 2, ¶2 “a series of parallel branches, each of which consists of a sequence of convolutional layers” and Fig. 4 – (b)).
Regarding the limitation and an intermediate channel between the postfix layer and the parallel operation layer is larger than an output channel from the postfix layer, Guo teaches and an intermediate channel between the postfix layer and the low dimensional layer is larger than an output channel from the postfix layer (Page 4, ¶1 “we therefore propose to expand a k × k convolutional layer into 3 consecutive convolutional layers: a 1×1 convolution; a k×k one; and another 1×1 one… this allows us to increase… the number of channels by setting p,q > n,m… for an original layer with m input channels and n output ones, given an expansion rate r, we define… the number of output channels of the intermediate k × k layer as q = rn,” Page 5, Section 4.1.1, ¶1, wherein the number of channels after the intermediate k × k convolutional layer, or “q = rn,” is greater than the number of output channels, or “n,” hence an intermediate channel between the postfix layer and the low dimensional layer, or “q,” is larger than an output channel from the postfix layer, or “n”). However, the combination of Guo and Kingman fails to teach …the parallel operation layer…
Hu teaches the parallel operation layer (Page 4, Col. 2, ¶2 “a series of parallel branches, each of which consists of a sequence of convolutional layers” and Fig. 4 – (b)).
Guo and Hu are analogous art to the claimed invention as both are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the trainable parallel operation layers of Hu with the prefix and postfix layers and the channel architecture of Guo. The motivation to do so is “to save the training-time memory cost by about 70% and accelerate the training speed by around 2×” (Hu, Abstract).
Regarding claim 7, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Hu teaches wherein the parallel operation layers are of a same size (Page 4, Col. 1, Section 3.3, ¶1 “we describe the standard procedure for squeezing a training-time linear block into a single convolutional kernel,” Col. 2, Fig. 4 – (b) and ¶1 “we reduce the extra training cost of re-param from O(H × W) to O(KH × KW) in terms of both computation and memory, where (H,W), (KH,KW) are the spatial shapes of the feature map and the convolutional kernel,” ¶2 “no matter how complicated a linear re-param block is, the following two properties always hold… The block can be represented by a series of parallel branches, each of which consists of a sequence of convolutional layers,” ¶3 “With the above two properties, we can squeeze a block if we can simplify… a multi-branch (i.e., the parallel structure) into a single convolution,” Page 5, Col. 1, ¶3 “The simplification of a parallel structure is trivial… we can merge multiple branches into one according to Eq. (4).
PNG
media_image1.png
92
531
media_image1.png
Greyscale
where Wm is the weight of the mth branch… when merging kernels of different sizes, we need to align the spatial centers of different kernels, e.g., an 1×1 kernel should be aligned with the center of a 3×3 kernel,” Page 11, Section 1.1, ¶1 “The pixel-wise form of convolution Y = W * X is:
PNG
media_image2.png
100
990
media_image2.png
Greyscale
Similarly, the pixel-wise form of convolution between two kernels… is defined in Eq. (2).
PNG
media_image3.png
334
1179
media_image3.png
Greyscale
where Wj… and Wj+1… are weights of two sequential convolutional layers, and Pad… means zero padding the weight tensor spatially from the left, right, top, and bottom,” wherein the “zero padding” operation to ensure that parallel operation layers are of a same size is implied to occur during “convolution between two kernels” of different sizes where spatial centers of different kernels need to be aligned, Page 4, Col. 2, Fig. 4 – (b) depicts the centering and padding operations).
Guo and Hu are analogous art to the claimed invention as both are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the generating of parallel operation layers of Hu with the high dimensional layer of Guo. The motivation to do so is “to save the training-time memory cost by about 70% and accelerate the training speed by around 2×” (Hu, Abstract).
Regarding claim 9, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Guo teaches wherein the low-dimensional layer is a convolution layer, an elementwise operation layer, or a scaling layer (Page 2, Fig. 1 and Page 4, ¶1 “a k×k one”), and the high-dimensional layer is a convolution layer, an elementwise operation layer, or a scaling layer (Page 2, Fig. 1 and ¶2 “replacing a k×k convolution by three convolutional layers with kernel size 1×1, k ×k and 1×1, respectively”).
Claims 10-17 and 19 are method claims with similar limitations to those of the methods of claims 1-7 and 9, respectively. Claims 10-17 and 19 are rejected under substantially the same rationale as set forth above with respect to claims 1-7 and 9, respectively.
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Kingman and further in view of Hu, and further in view of Zagoruyko et al. (“DiracNets: Training Very Deep Neural Networks without Skip-Connections,” 2018, hereinafter Zagoruyko).
Regarding claim 8, Guo in view of Kingman and further in view of Hu teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Regarding the limitation wherein assigning the initial weights to the parallel operation layers is performed according to an arbitrary probability distribution, Hu teaches the parallel operation layers (Page 4, Col. 2, ¶2 “a series of parallel branches, each of which consists of a sequence of convolutional layers”). However, the combination of Guo, Kingman, and Hu fails to teach wherein assigning the initial weights to the parallel operation layers is performed according to an arbitrary probability distribution.
Zagoruyko, in the same field of endeavor, teaches wherein assigning the initial weights to a layer is performed according to an arbitrary probability distribution (Page 3, ¶1 “a simple linear layer W ⊙ x,” ¶2 “We also use weight normalization… for W… We initialize W from normal distribution N(0,1)”).
Guo, Hu, and Zagoruyko are analogous art to the claimed invention as all are from the same field of endeavor of convolutional neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the weight initialization of Zagoruyko and the parallel operation layers of Hu with the high-dimensional layer and methodology of Guo. The motivation to do so is “to save the training-time memory cost by about 70% and accelerate the training speed by around 2×” (Hu, Abstract) and to eliminate “the need of careful initialization in residual and non-residual networks” (Zagoruyko, Abstract).
Claim 18 is a method claim with similar limitations to the method of claim 8. Claim 18 is rejected under substantially the same rationale as set forth above with respect to claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM M LEE whose telephone number is (571)272-4761. The examiner can normally be reached Mon-Fri. 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM MICHAEL LEE/
Examiner, Art Unit 2145
/CHAU T NGUYEN/Primary Examiner, Art Unit 2145
Prosecution Insights