DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Claim 1 recites the following contingent limitation(s): identifying whether one or more layers of the DNN that satisfy at least one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers. The limitation(s) is/are contingent because a determination could be made for none of these conditions, or 1-3 conditions. The BRI of the claim requires an interpretation that none of the conditions are determined to exist, therefore requiring none of the subsequent limitations to be performed, including all dependent claims. See MPEP 2111.04. This also applies to claim 9.
Claim Objections
The following claims are objected to because of the following informalities:
Claim 1 recites identifying whether one or more layers of the DNN that satisfy at least one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers, but it appears to be Applicant’s intention to avoid the above claim interpretation. Therefore, this limitation should be amended to recite identifying This phrasing requires one, two, or three of the conditions, but removes avoiding all three conditions. This also applies to claim 9.
Appropriate correction is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-4, 6-12, 14-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Exemplary claim 1 in line 3 recites identifying whether one or more layers of the DNN that satisfy at least one of a first, a second and a third condition. The use of the terms “whether” and “that” together render the claim indefinite. It is unclear whether the scope of the claim is 1) identifying whether one of the layers that is now satisfying one of the conditions, or 2) identifying whether one of the layers that is already satisfying one of the conditions. In essence, the “whether” directs the scope to identify if a layer satisfies a condition, but the “that” directs the scope to identify from a group that does satisfy a condition.
For this reason, the above listed claims are rejected for containing this language or being dependent on a claim that contains this language.
Claims 1-4, 6-12, 14-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Exemplary claim 1 in line 14 recites combining the one or more convolution layers. It is unclear how one convolution layer can be combined.
For this reason, the above listed claims are rejected for containing this language or being dependent on a claim that contains this language.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4, 6-12, 14-16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 1 is a method claim. Claim 9 is a system claim. Therefore, claims 1 and 9 are directed to either a process, machine, manufacture or composition of matter.
With respect to Claim 1:
Step 2A Prong 1:
identifying whether one or more layers of the DNN that satisfy at least one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers, wherein: the first condition includes whether the one or more convolutional layers are placed in one or more parallel branches of the DNN; the second condition includes whether at least one of the resampling layers has a specified first resampling ratio; and the third condition includes whether at least one of the resampling layers is followed by a convolution layer (mental process – user can manually determine whether one or more layers of the DNN satisfy one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers, wherein: the first condition includes whether the one or more convolutional layers are placed in one or more parallel branches of the DNN; the second condition includes whether at least one of the resampling layers has a specified first resampling ratio; and the third condition includes whether at least one of the resampling layers is followed by a convolution layer)
performing the on-device inference based on the identified one or more layers, wherein performing the on-device inference comprises at least one of: combining the one or more convolution layers placed in the one or more parallel branches, based on the one or more layers of the DNN satisfying the first condition; optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition; and modifying operation of the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the third condition (mental process – user can manually perform the inference based on the determination, wherein performing the inference comprises at least one of: optimizing the one or more convolution layers in the one or more parallel branches, based on the one or more layers of the DNN satisfying the first condition; optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition; and modifying operation of the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the third condition)
wherein optimizing the at least one of the resampling layers comprises: cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio; adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio; modifying an inference graph based on the plurality of cascaded resampling layers and the added convolution layer (mental process – user can manually doing at least one of cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio; adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio; or modifying an inference graph based on the plurality of cascaded resampling layers and the added convolution layer)
Step 2A Prong 2: This judicial exception is not integrated into a practical application. Additional elements:
on-device (mere instructions to apply the exception using a generic computer component)
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception. Additional elements:
on-device (mere instructions to apply the exception using a generic computer component)
Conclusion: The claim is not patent eligible.
Claims 9 is rejected on the same grounds as claim 1. Claim 9 has the additional elements of a memory and a processor coupled to the memory. These elements are mere instructions to apply the exception using a generic computer component under Step 2A prong 2 and Step 2B. These additional elements and accompanying analysis apply to claims 10-16 which depend on claim 9.
Regarding Claim 2: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually computing a number of channels required for a preceding convolution layer based on the received graph, wherein the preceding convolution layer is preceding to the one or more convolution layers;
adding a number of filters in the preceding convolution layer by adding a plurality of dummy weights; and
combining the one or more convolution layers placed in the one or more parallel branches into one convolution layer based on the added number of filters in the preceding convolution layer.
The limitation(s) includes the additional elements of receiving an inference graph.
These judicial exceptions are not integrated into a practical application. The additional element(s) of receiving an inference graph recite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of receiving an inference graph recite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claims are not patent eligible.
Regarding Claim 3: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually selecting a common kernel size for the combined convolution layer, wherein the kernel size is greater than or equal to kernel size of each convolution layer of the one or more convolutional layers;
computing a number of first filters required in each convolution layer of each of the one or more parallel branches;
computing a number of second filters required in the combined convolution layer, wherein the number of second filters is equal to a product of the number of first filters in each convolution layer and a number of the one or more parallel branches;
adjusting a plurality of weights for the first filters in the one or more parallel branches based on a number of extra filters added in the preceding convolution layer and the number of second filters required in the combined convolution layer;
re-arranging the plurality of adjusted weights for filters in the combined convolution layer; and
modifying the inference graph based on the re-arranging.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 4: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually modifying the number of extra filters in the preceding convolution layer such that the number of extra filters is equal to a number of channels of the one or more convolution layers after concatenation, wherein the preceding convolution layer is preceding to the one or more convolution layers; and
modifying a number of other filters in the one or more convolution layers based on the modified number of extra filters in the preceding convolution layer.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 6: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually modifying weights for the convolution layer;
interleaving a plurality of dummy weights with the modified weights in filters of the convolutional layer;
respacing the interleaved weights in filters of the convolutional layer; and
performing a dimension scaling operation on an inference graph.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 7: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually wherein the one or more resampling layers includes one or more depth to space layers or one or more transpose convolution layers.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 8: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually wherein the specified first resampling ratio is greater than or equal to.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Claims 10-12, 14-16 are rejected on the same grounds as Claims 2-4, 6-8 respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 8, 9, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over A et al. (hereinafter Yong), U.S. Patent Application Publication 2020/0311552 in view of Thoma, Analysis and Optimization of Convolutional Neural Network Architectures.
Regarding Claim 1, Yong discloses a method for on-device inference in a deep neural network (DNN) [“With the popularity of portable devices such as mobile phones, there is an increasing demand for operating neural network models on a device side.” ¶3; “the machine learning model may comprise a deep neural network structure, a convolution neural network, a recurrent neural network, or a combination thereof.” ¶85], the method comprising:
identifying whether one or more layers of the DNN that satisfy at least one of a first, a second and a third condition [“in a neural network, the influence on the size and operating speed of the model is concentrated in a network layer such as a convolutional layer or a fully connected layer” ¶59; Examiner Note: the structure of a neural network greatly impacts the resources required; Fig. 6; Examiner Note: Figure 6 shows a deep neural network and determining the structure would determine the conditions], the one or more layers including one or more convolution layers and one or more resampling layers [Fig. 6; Examiner Note: Figure 6 shows one or more convolution layers (i.e., convolution) and one or more resampling layers (i.e., pooling layer)], wherein:
the first condition includes whether the one or more convolutional layers are placed in one or more parallel branches of the DNN [Fig. 6; Examiner Note: Figure 6 shows multiple convolution layers in parallel branches];
the second condition includes whether at least one of the resampling layers has a specified first resampling ratio; and
the third condition includes whether at least one of the resampling layers is followed by a convolution layer [Fig. 6; Examiner Note: the left most branch of the DNN shown has a convolution layer subsequent to a resampling layer]; and
performing the on-device inference based on the identified one or more layers [“With the popularity of portable devices such as mobile phones, there is an increasing demand for operating neural network models on a device side.” ¶3], wherein performing the on-device inference comprises at least one of:
combining the one or more convolution layers placed in the one or more parallel branches, based on the one or more layers of the DNN satisfying the first condition [Fig. 6; Examiner Note: The Concatenate layer combines multiple convolution layers];
optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition; and
modifying operation of the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the third condition [“The machine learning model may correspond to an artificial intelligence model, and may be a model generated by machine learning based on training data.” ¶83; Examiner Note: Training is one way to modify operation of a model],
wherein optimizing the at least one of the resampling layers comprises:
cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio;
adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio; and
modifying an inference graph based on the plurality of cascaded resampling layers and the added convolution layer [“The model may be defined by nodes, branches between the nodes, at least one layer, functions of the at least one layer, and weight values of the branches.” ¶83; Fig. 6].
However, Yong fails to explicitly disclose the second condition includes whether at least one of the resampling layers has a specified first resampling ratio;
optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition; and
wherein optimizing the at least one of the resampling layers comprises:
cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio;
adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio.
Thoma discloses the second condition includes whether at least one of the resampling layers has a specified first resampling ratio [“Pooling summarizes a p x p area of the input feature map. Just like convolutional layers, pooling can be used with a stride of s є N>1. As s ≥ 2 is the usual choice, pooling layers are sometimes also called subsampling layers. Typically, p є { 2, 3, 4, 5} and s = 2 such as for AlexNet [KSH12] and VGG-16 [SZ14].” §2.2.2];
optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition [“Convolutional Neural Networks (CNNs) dominate various computer vision tasks since Alex Krizhevsky showed that they can be trained effectively” Abstract; Table 5.1 and Figure 5.1; Examiner Note: Table 5.1 displays pooling at layer 4 of a ratio of 2x2, at least one intervening convolution layer, for example at layer 5, and pooling again at layer 15 of a ratio of 1x1 which is less than the previous ratio.]; and
wherein optimizing the at least one of the resampling layers comprises:
cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio; adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio [Table 5.1 and Figure 5.1; Examiner Note: Table 5.1 displays pooling at layer 4 of a ratio of 2x2, at least one intervening convolution layer, for example at layer 5, and pooling again at layer 15 of a ratio of 1x1 which is less than the previous ratio.].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the method of Yong to incorporate the resampling ratio and cascading resampling layers with decreasing ratios and an intervening convolutional layer of Thoma.
Given the advantage of optimizing resampling layers to improve efficiency and to provide local translational invariance, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 8, Yong and Thoma disclose the method as claimed in claim 1.
However, Yong fails to explicitly disclose wherein the specified first resampling ratio is greater than or equal to 5.
Thoma discloses wherein the specified first resampling ratio is greater than or equal to 5 [“Pooling summarizes a p x p area of the input feature map. Just like convolutional layers, pooling can be used with a stride of s є N>1. As s ≥ 2 is the usual choice, pooling layers are sometimes also called subsampling layers. Typically, p є { 2, 3, 4, 5} and s = 2 such as for AlexNet [KSH12] and VGG-16 [SZ14].” §2.2.2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the combination to incorporate a ratio greater than or equal to 5 of Thoma.
Given the advantage of maximizing efficiency, reduce cost, and speed up processing, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claims 9, 16 are rejected on the same grounds as claims 1, 8 respectively.
Claim(s) 2-4, 10-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yong and Thoma, in view of Ding et al. (hereinafter Ding), Diverse Branch Block: Building a Convolution as an Inception-like Unit.
Regarding Claim 2, Yong and Thoma disclose the method as claimed in claim 1. Yong further discloses wherein combining the one or more convolution layers comprises: receiving an inference graph [“The model may be defined by nodes, branches between the nodes, at least one layer, functions of the at least one layer, and weight values of the branches.” ¶83; Fig. 6].
However, Yong fails to explicitly disclose computing a number of channels required for a preceding convolution layer based on the received graph, wherein the preceding convolution layer is preceding to the one or more convolution layers;
adding a number of filters in the preceding convolution layer by adding a plurality of dummy weights.
Thoma discloses computing a number of channels required for a preceding convolution layer based on the received graph, wherein the preceding convolution layer is preceding to the one or more convolution layers [“A linear image filter (also called a filter bank or a kernel) is an element F є Rkwxkhxd, where kw represents the filter’s width, kh the filter’s height and d the number of input channels. The filter F is convolved with the image I є Rwxhxd to produce a new image Il” ¶2.1 ¶1.];
adding a number of filters in the preceding convolution layer by adding a plurality of dummy weights [“adding more filters to that layer could improve the performance” §2.5.8 ¶2; “with zero padding at the borders” §2.5.8 ¶3].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the combination to incorporate the channel and filter usage of Thoma.
Given the advantage of determining channels and adding filters for convolution, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Yong fails to explicitly disclose combining the one or more convolution layers placed in the one or more parallel branches into one convolution layer based on the added number of filters in the preceding convolution layer.
Ding discloses combining the one or more convolution layers placed in the one or more parallel branches into one convolution layer based on the added number of filters in the preceding convolution layer [“Transform VI: a conv for multi-scale convolutions Considering a kh × kw (kh _ K, kw _ K) kernel is equivalent to a K × K kernel with some zero entries, we can transform a kh × kw kernel into K × K via zero-padding. Specifically, 1 × 1, 1 × K and K × 1 conv are particularly practical as they can be efficiently implemented. The input should be padded to align the sliding windows (Fig. 4).” §3.2 § Transform VI; Fig. 2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Ding before him before the effective filing date of the claimed invention, to modify the combination to incorporate the combining of convolution layers of Ding.
Given the advantage of consolidating the neural network to run on hardware constrained systems, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 3, Yong, Thoma, and Ding disclose the method as claimed in claim 2. Yong further discloses further comprising: modifying the inference graph based on the re-arranging [“The model may be defined by nodes, branches between the nodes, at least one layer, functions of the at least one layer, and weight values of the branches.” ¶83; Fig. 6].
However, Yong fails to explicitly disclose selecting a common kernel size for the combined convolution layer, wherein the kernel size is greater than or equal to kernel size of each convolution layer of the one or more convolutional layers;
computing a number of first filters required in each convolution layer of each of the one or more parallel branches;
computing a number of second filters required in the combined convolution layer, wherein the number of second filters is equal to a product of the number of first filters in each convolution layer and a number of the one or more parallel branches;
adjusting a plurality of weights for the first filters in the one or more parallel branches based on a number of extra filters added in the preceding convolution layer and the number of second filters required in the combined convolution layer;
re-arranging the plurality of adjusted weights for filters in the combined convolution layer.
Ding discloses selecting a common kernel size for the combined convolution layer, wherein the kernel size is greater than or equal to kernel size of each convolution layer of the one or more convolutional layers [“KxK” Figure 2];
computing a number of first filters required in each convolution layer of each of the one or more parallel branches [“Kx1” and “1xK” Figure 2; “The parameters of a conv layer with C input channels, D output channels and kernel size K × K reside in the conv kernel, which is a 4th-order tensor F є RD×C×K×K, and an optional bias b є RD. It takes a C-channel feature map I є RC×H×W as input and outputs a D-channel feature map O є RD×H′ ×W′ , where H′ and W′ are determined by K, padding and stride configurations.” §3.1];
computing a number of second filters required in the combined convolution layer, wherein the number of second filters is equal to a product of the number of first filters in each convolution layer and a number of the one or more parallel branches [“KxK” Figure 2; “The parameters of a conv layer with C input channels, D output channels and kernel size K × K reside in the conv kernel, which is a 4th-order tensor F є RD×C×K×K, and an optional bias b є RD. It takes a C-channel feature map I є RC×H×W as input and outputs a D-channel feature map O є RD×H′ ×W′ , where H′ and W′ are determined by K, padding and stride configurations.” §3.1];
adjusting a plurality of weights for the first filters in the one or more parallel branches based on the number of extra filters added in the preceding convolution layer and the number of second filters required in the combined convolution layer [“Kx1” and “1xK” Figure 2; “an optional bias b є RD” §3.1; “data-dependent kernel re-parameterization, as it generated the weights for multiple kernels of the same shape, then derived a kernel as the weighted sum of all such kernels to participate in the convolution” §2.4];
re-arranging the plurality of adjusted weights for filters in the combined convolution layer [“KxK” Figure 2; “an optional bias b є RD” §3.1; “data-dependent kernel re-parameterization, as it generated the weights for multiple kernels of the same shape, then derived a kernel as the weighted sum of all such kernels to participate in the convolution” §2.4].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Ding before him before the effective filing date of the claimed invention, to modify the combination to incorporate the combining of convolution layers of Ding.
Given the advantage of consolidating the neural network to run on hardware constrained systems, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 4, Yong, Thoma, and Ding disclose the method as claimed in claim 3. Yong further discloses wherein adjusting the plurality of weights comprises:
modifying the number of extra filters in the preceding convolution layer such that the number of extra filters is equal to a number of channels of the one or more convolution layers after concatenation, wherein the preceding convolution layer is preceding to the one or more convolution layers [“a convolutional layer, as shown in FIG. 2A, the left side is a group of input feature maps (total c,=4 feature maps), and the right side is a group of output feature maps ( c,+i =5 feature maps). Assuming that the width and height of the input feature map are W, and H,, respectively, and the width and height of the output feature map are W,+i and H,+i, respectively. The convolutional layer may contain 5 filters, which are also known as convolution kernels, and each filter may correspond to an output feature map and may contain 4 kernels (representing a two-dimensional kernel, i.e., a two-dimensional part of the convolution kernel). The width and height of a kernel are usually referred to as kernel size (k(i)wxk(i)h, for example lxl, 3x3, 5x5, etc.).” ¶104; Examiner Note: Filters/Kernels are based on the size of the feature maps];
modifying a number of other filters in the one or more convolution layers based on the modified number of extra filters in the preceding convolution layer [“a convolutional layer, as shown in FIG. 2A, the left side is a group of input feature maps (total c,=4 feature maps), and the right side is a group of output feature maps ( c,+i =5 feature maps). Assuming that the width and height of the input feature map are W, and H,, respectively, and the width and height of the output feature map are W,+i and H,+i, respectively. The convolutional layer may contain 5 filters, which are also known as convolution kernels, and each filter may correspond to an output feature map and may contain 4 kernels (representing a two-dimensional kernel, i.e., a two-dimensional part of the convolution kernel). The width and height of a kernel are usually referred to as kernel size (k(i)wxk(i)h, for example lxl, 3x3, 5x5, etc.).” ¶104; Examiner Note: Filters/Kernels are based on the size of the feature maps].
Claims 10-12 are rejected on the same grounds as claims 2-4 respectively.
Claim(s) 6, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yong and Thoma, in view of Liu et al. (hereinafter Liu), Sparse Convolutional Neural Networks, further in view of Mehta, PermNet: Permuted Convolutional Neural Network.
Regarding Claim 6, Yong and Thoma disclose the method as claimed in claim 1.
However, Yong fails to explicitly disclose wherein, based on the at least one of the resampling layers being followed by the convolution layer, performing the on-device inference comprises:
modifying weights for the convolution layer;
interleaving a plurality of dummy weights with the modified weights in filters of the convolutional layer.
Thoma discloses wherein, based on the at least one of the resampling layers being followed by the convolution layer, performing the inference comprises:
modifying weights for the convolution layer [“Figures 5.8 to 5.10 show how the weights changed while training” §5.1.3 ¶3];
interleaving a plurality of dummy weights with the modified weights in filters of the convolutional layer [“with zero padding at the borders” §2.5.8 ¶3].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the combination to incorporate weight modification and dummy weights of Thoma.
Given the advantage of more accurate models, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Yong fails to explicitly disclose respacing the interleaved weights in filters of the convolutional layer.
Liu discloses respacing the interleaved weights in filters of the convolutional layer [“An example sparse matrix B. The shadowed squares represent non-zero elements and the blank squares represent zero elements” Fig 3(a)].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Liu before him before the effective filing date of the claimed invention, to modify the combination to incorporate a sparse matrix of Liu.
Given the advantage of faster processing, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Yong fails to explicitly disclose performing a dimension scaling operation on an inference graph.
Mehta discloses performing a dimension scaling operation on an inference graph [“Model scaling. (a) The baseline network to be scaled. (b-d) Networks with width, depth and input image resolution scaled respectively.” Fig. 1.1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, Liu, and Mehta before him before the effective filing date of the claimed invention, to modify the combination to incorporate the scaling choices of Mehta.
Given the advantage of decreasing or increasing network complexity, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 14 is rejected on the same grounds as claim 6.
Claim(s) 7, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yong and Thoma, in view of Du et al. (hereinafter Du), License plate super-resolution reconstruction based on improved ESPCN network.
Regarding Claim 7, Yong and Thoma disclose the method as claimed in claim 1.
However, Yong fails to explicitly disclose wherein the one or more resampling layers includes one or more depth to space layers or one or more transpose convolution layers.
Du discloses wherein the one or more resampling layers includes one or more depth to space layers or one or more transpose convolution layers [“convolutional layer and the depth to space layer” §II.B ¶1; Fig. 3].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Du before him before the effective filing date of the claimed invention, to modify the combination to incorporate the depth to space layer of Du.
Given the advantage of upsampling to reconstruct images with accurate boundaries, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 15 is rejected on the same grounds as claim 7.
Examiner’s Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification.
Response to Arguments
Regarding the §101 rejections, Applicant's arguments have been fully considered but have been found unpersuasive. Applicant argues that 1) the claimed invention cannot be practically performed in the mind, 2) the claim invention provides concrete technical improvements, 3) the claims integrate the judicial exception into a practical application, and 4) the claim elements as a whole amount to significantly more than the judicial exception. Examiner disagrees for at least the following reasons.
First, the invention as claimed can be performed in the mind or with pencil and paper. Applicant’s arguments are narrower than the broadest reasonable interpretation of the claims that the current language permits. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. In reVan Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Exemplary claim 1 can be broken down into essentially two steps. The first step is identifying at least one of three conditions based on the design of the neural network. This can be done by a human mentally through observation. The second step is altering the design of the neural network in at least one of three ways. Again, this design step can be done mentally using pencil and paper. The instant amendments are only to one of the three altering steps, so under broadest reasonable interpretation, these amendments are not implemented since another step can be used instead. Therefore, the claim, with exception of the additional element, can be practically performed in the mind.
Second, the claims do not provide a concrete technical improvement. Instead, any alleged improvement is to the abstract idea. Applicant asserts that the reduced computational complexity leads to improved computer functionality (i.e., less resources used). However, the computer functions the same. Rather it is the neural network design which is allegedly less complex. Accordingly, the claims do not provide a technical improvement.
Third, the additional elements do not integrate the judicial exception into a practical application. Applicant asserts the claim include a combination of additional elements which impose meaningful limitations on the judicial exception. However, Applicant fails to indicate specifically what those additional elements are. As explained in the rejection, the only additional element in claim 1 is the recitation of the method being for an on-device inference. As a result, the additional elements do not integrate the judicial exception into a practical application
Fourth, the claim as a whole does not amount to significantly more than the judicial exception. Applicant supports their assertion to the contrary by relying on the argument that the prior art fails to show the combination of elements, so that combination is therefore not well-understood, routine, or convention activities. Examiner refers Applicant to MPEP 2106 which states, "The question of whether a particular claimed invention is novel or obvious is "fully apart" from the question of whether it is eligible. Diamond v. Diehr, 450 U.S. 175, 190, 209 USPQ 1,9 (1981)." This can be further supported by SAP America v Investpic, a precedential case by the Federal Circuit. In this case the court states:
We may assume that the techniques claimed are “[g]roundbreaking, innovative, or even brilliant,” but that is not enough for eligibility. Ass’n for Molecular Pathology v. Myriad Genetics, Inc., 569 U.S. 576, 591 (2013); accord buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1352 (Fed. Cir. 2014). Nor is it enough for subject-matter eligibility that claimed techniques be novel and nonobvious in light of prior art, passing muster under 35 U.S.C. §§ 102 and 103. See Mayo Collaborative Servs. v. Prometheus Labs., Inc., 566 U.S. 66, 89-90 (2012); Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1151 (Fed. Cir. 2016) (“[ A] claim for a new abstract idea is still an abstract idea. The search for a § 101 inventive concept is thus distinct from demonstrating § 102 novelty.”); Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307 ,1315 (Fed. Cir. 2016) (same for obviousness) (Symantec). The claims here are ineligible because their innovation is an innovation in ineligible subject matter.
Likewise, the instant claims recite a mathematical arrangement for calculations (i.e. a neural network design) which can be determined mentally by a person. The variation in connections between the layers is functionally nothing more than variations in the calculations of data. Consequently, the claim as a whole does not amount to significantly more than the judicial exception.
For at least these reasons, the §101 rejections are maintained.
Regarding the prior art rejections, Applicant's arguments have been fully considered but have been found unpersuasive. Applicant argues that 1) the combination of Yong and Thoma does not disclose optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition and wherein optimizing the at least one of the resampling layers comprises: cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio; adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio; and modifying an inference graph based on the plurality of cascaded resampling layers and the added convolution layer, and 2) a reasoned basis for combination was not provided. Examiner disagrees for at least the following reasons.
First, Applicant is arguing against one of three possible conditions, which under broadest reasonable interpretation are recited in the alternative. Accordingly, even if Examiner agreed with Applicant, which is not the case, the other two conditions are still rejected thereby rejecting the whole claim. Therefore, Applicant’s argument is not persuasive. However, for the sake of compact prosecution, Applicant’s argument will still be briefly addressed. Yong discloses a resampling layer (i.e., pooling layer in at least Fig. 6) and Thoma discloses a design having a resampling layer then convolution layer then resampling layer with the resampling layers downsampling as shown in at least Table 5.1 and Fig. 5.1. The combination of references involves substituting the resampling layer of Yong with the architecture design of Thoma.
Second, a motivation to combine the references was provided in the rejection. As an initial matter, both Yong and Thoma are concerned with neural network optimization through design alterations, which is the same as the instant invention. While explicit disclosures of motivation are often used, when “considering the disclosure of a reference, it is proper to take into account not only specific teachings of the reference but also the inferences which one skilled in the art would reasonably be expected to draw therefrom." In re Preda, 401 F.2d 825, 826, 159 USPQ 342, 344 (CCPA 1968), MPEP2144.01. Furthermore, “there is no requirement that an ‘express, written motivation to combine must appear in prior art references before a finding of obviousness’” MPEP 2145. Accordingly, optimizing resampling layers in a neural network, such as that found in the experimental evaluation of Thoma, would motivate a person having ordinary skill in the art to combine Yong and Thoma by replacing a resampling layer of Yong (ie, pooling layer) with the disclosed design of Thoma having a resampling layer then convolution layer then resampling layer with the resampling layers downsampling.
For at least these reasons, the rejections are maintained.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
The following references were found during the examination of this patent application and were found to be relevant to patentability. Applicant is advised to review these references prior to responding to this Office action.
Gildenblat (Accelerating Deep Neural Networks with Tensor Decompositions) discloses a method to take a layer and decompose it into several smaller layers.
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.B./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights