DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Abstract
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
With the exception of the first phrases which can be implied (“The disclosure relates to”), the abstract is identical in form and language to claim 1. Accordingly, the abstract is objected to for not satisfying the requirements of MPEP 608.01(b).
Title
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. Examiner believes that the title of the invention is imprecise. A descriptive title indicative of the invention will help in proper indexing, classifying, searching, etc. See MPEP 606.01. However, the title of the invention should be limited to 500 characters. Examiner suggests including the aspect(s) of the claims which Applicant believes to be novel or nonobvious over the prior art.
Claim Interpretation
Claim 1 recites the following contingent limitation(s): determining whether one or more layers of the DNN satisfy one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers. The limitation(s) is/are contingent because a determination could be made for none of these conditions, or 1-3 conditions. The BRI of the claim requires an interpretation that none of the conditions are determined to exist, therefore requiring none of the subsequent limitations to be performed, including all dependent claims. See MPEP 2111.04.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 4, 6, 11, 12, and 14 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 3, 4, 11, and 12 each recite, the number of extra filters without proper antecedent basis. Claims 6 and 14 each recite, the inference without proper antecedent basis.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 1 is a method claim. Claim 9 is a system claim. Therefore, claims 1 and 9 are directed to either a process, machine, manufacture or composition of matter.
With respect to Claim 1:
Step 2A Prong 1:
determining whether one or more layers of the DNN satisfy one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers, wherein: the first condition includes whether the one or more convolutional layers are placed in one or more parallel branches of the DNN; the second condition includes whether at least one of the resampling layers has a specified first resampling ratio; and the third condition includes whether at least one of the resampling layers is followed by a convolution layer (mental process – user can manually determine whether one or more layers of the DNN satisfy one of a first, a second and a third condition, the one or more layers including one or more convolution layers and one or more resampling layers, wherein: the first condition includes whether the one or more convolutional layers are placed in one or more parallel branches of the DNN; the second condition includes whether at least one of the resampling layers has a specified first resampling ratio; and the third condition includes whether at least one of the resampling layers is followed by a convolution layer)
performing the on-device inference based on the determination, wherein performing the on-device inference comprises at least one of: optimizing the one or more convolution layers in the one or more parallel branches, based on the one or more layers of the DNN satisfying the first condition; optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition; and modifying operation of the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the third condition (mental process – user can manually perform the inference based on the determination, wherein performing the inference comprises at least one of: optimizing the one or more convolution layers in the one or more parallel branches, based on the one or more layers of the DNN satisfying the first condition; optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition; and modifying operation of the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the third condition)
Step 2A Prong 2: This judicial exception is not integrated into a practical application. Additional elements:
on-device (mere instructions to apply the exception using a generic computer component)
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception. Additional elements:
on-device (mere instructions to apply the exception using a generic computer component)
Conclusion: The claim is not patent eligible.
Claims 9 is rejected on the same grounds as claim 1. Claim 9 has the additional elements of a memory and a processor coupled to the memory. These elements are mere instructions to apply the exception using a generic computer component under Step 2A prong 2 and Step 2B. These additional elements and accompanying analysis apply to claims 10-16 which depend on claim 9.
Regarding Claim 2: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually computing a number of channels required for a preceding convolution layer based on the received graph, wherein the preceding convolution layer is preceding to the one or more convolution layers;
adding a number of filters in the preceding convolution layer by adding a plurality of dummy weights; and
combining the one or more convolution layers placed in the one or more parallel branches into one convolution layer based on the added number of filters in the preceding convolution layer.
The limitation(s) includes the additional elements of receiving an inference graph.
These judicial exceptions are not integrated into a practical application. The additional element(s) of receiving an inference graph recite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of receiving an inference graph recite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claims are not patent eligible.
Regarding Claim 3: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually selecting a common kernel size for the combined convolution layer, wherein the kernel size is greater than or equal to kernel size of each convolution layer of the one or more convolutional layers;
computing a number of first filters required in each convolution layer of each of the one or more parallel branches;
computing a number of second filters required in the combined convolution layer, wherein the number of second filters is equal to a product of the number of first filters in each convolution layer and a number of the one or more parallel branches;
adjusting a plurality of weights for the first filters in the one or more parallel branches based on the number of extra filters added in the preceding convolution layer and the number of second filters required in the combined convolution layer;
re-arranging the plurality of adjusted weights for filters in the combined convolution layer; and
modifying the inference graph based on the re-arranging.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 4: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually modifying the number of extra filters in the preceding convolution layer such that the number of extra filters is equal to a number of channels of the one or more convolution layers after concatenation, wherein the preceding convolution layer is preceding to the one or more convolution layers; and
modifying a number of other filters in the one or more convolution layers based on the modified number of extra filters in the preceding convolution layer.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 5: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio;
adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio; and
modifying an inference graph based on the plurality of cascaded resampling layers and the added convolution layer.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 6: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually modifying weights for the convolution layer;
interleaving a plurality of dummy weights with the modified weights in filters of the convolutional layer;
respacing the interleaved weights in filters of the convolutional layer; and
performing a dimension scaling operation on an inference graph.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 7: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually wherein the one or more resampling layers includes one or more depth to space layers or one or more transpose convolution layers.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claim 8: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind.
The limitation(s) encompasses the user manually wherein the specified first resampling ratio is greater than or equal to.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Claims 10-16 are rejected on the same grounds as Claims 2-8 respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5, 8, 9, 13, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over A et al. (hereinafter Yong), U.S. Patent Application Publication 2020/0311552 in view of Thoma, Analysis and Optimization of Convolutional Neural Network Architectures.
Regarding Claim 1, Yong discloses a method for on-device inference in a deep neural network (DNN) [“With the popularity of portable devices such as mobile phones, there is an increasing demand for operating neural network models on a device side.” ¶3; “the machine learning model may comprise a deep neural network structure, a convolution neural network, a recurrent neural network, or a combination thereof.” ¶85], the method comprising:
determining whether one or more layers of the DNN satisfy one of a first, a second and a third condition [“in a neural network, the influence on the size and operating speed of the model is concentrated in a network layer such as a convolutional layer or a fully connected layer” ¶59; Examiner Note: the structure of a neural network greatly impacts the resources required; Fig. 6; Examiner Note: Figure 6 shows a deep neural network and determining the structure would determine the conditions], the one or more layers including one or more convolution layers and one or more resampling layers [Fig. 6; Examiner Note: Figure 6 shows one or more convolution layers (i.e., convolution) and one or more resampling layers (i.e., pooling layer)], wherein:
the first condition includes whether the one or more convolutional layers are placed in one or more parallel branches of the DNN [Fig. 6; Examiner Note: Figure 6 shows multiple convolution layers in parallel branches];
the second condition includes whether at least one of the resampling layers has a specified first resampling ratio; and
the third condition includes whether at least one of the resampling layers is followed by a convolution layer [Fig. 6; Examiner Note: the left most branch of the DNN shown has a convolution layer subsequent to a resampling layer]; and
performing the on-device inference based on the determination [“With the popularity of portable devices such as mobile phones, there is an increasing demand for operating neural network models on a device side.” ¶3], wherein performing the on-device inference comprises at least one of:
optimizing the one or more convolution layers in the one or more parallel branches, based on the one or more layers of the DNN satisfying the first condition [“The machine learning model may correspond to an artificial intelligence model, and may be a model generated by machine learning based on training data.” ¶83; Examiner Note: Training is one way to optimize a model];
optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition ; and
modifying operation of the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the third condition [“The machine learning model may correspond to an artificial intelligence model, and may be a model generated by machine learning based on training data.” ¶83; Examiner Note: Training is one way to modify operation of a model].
However, Yong fails to explicitly disclose the second condition includes whether at least one of the resampling layers has a specified first resampling ratio;
optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition .
Thoma discloses the second condition includes whether at least one of the resampling layers has a specified first resampling ratio [“Pooling summarizes a p x p area of the input feature map. Just like convolutional layers, pooling can be used with a stride of s є N>1. As s ≥ 2 is the usual choice, pooling layers are sometimes also called subsampling layers. Typically, p є { 2, 3, 4, 5} and s = 2 such as for AlexNet [KSH12] and VGG-16 [SZ14].” §2.2.2];
optimizing the at least one of the resampling layers, based on the one or more layers of the DNN satisfying the second condition [“Convolutional Neural Networks (CNNs) dominate various computer vision tasks since Alex Krizhevsky showed that they can be trained effectively” Abstract; Table 5.1 and Figure 5.1; Examiner Note: Table 5.1 displays pooling at layer 4 of a ratio of 2x2, at least one intervening convolution layer, for example at layer 5, and pooling again at layer 15 of a ratio of 1x1 which is less than the previous ratio.].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the method of Yong to incorporate the resampling ratio of Thoma.
Given the advantage of optimizing resampling layers to improve efficiency, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 5, Yong and Thoma disclose the method as claimed in claim 1. Yong further discloses modifying an inference graph based on the plurality of cascaded resampling layers and the added convolution layer [“The model may be defined by nodes, branches between the nodes, at least one layer, functions of the at least one layer, and weight values of the branches.” ¶83; Fig. 6].
However, Yong fails to explicitly disclose wherein optimizing the at least one of the resampling layers comprises:
cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio;
adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio.
Thoma discloses wherein optimizing the at least one of the resampling layers comprises:
cascading the at least one of the resampling layers of the specified first resampling ratio into a plurality of cascaded resampling layers of a specified second resampling ratio, wherein the specified second resampling ratio is less than the specified first resampling ratio;
adding a convolution layer between two cascaded resampling layers among the plurality of cascaded resampling layers of the specified second resampling ratio [Table 5.1 and Figure 5.1; Examiner Note: Table 5.1 displays pooling at layer 4 of a ratio of 2x2, at least one intervening convolution layer, for example at layer 5, and pooling again at layer 15 of a ratio of 1x1 which is less than the previous ratio.].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the combination to incorporate cascading resampling layers with decreasing ratios and an intervening convolutional layer of Thoma.
Given the advantage of local translational invariance, to get invariance against minor local changes and for data reduction, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 8, Yong and Thoma disclose the method as claimed in claim 1.
However, Yong fails to explicitly disclose wherein the specified first resampling ratio is greater than or equal to 5.
Thoma discloses wherein the specified first resampling ratio is greater than or equal to 5 [“Pooling summarizes a p x p area of the input feature map. Just like convolutional layers, pooling can be used with a stride of s є N>1. As s ≥ 2 is the usual choice, pooling layers are sometimes also called subsampling layers. Typically, p є { 2, 3, 4, 5} and s = 2 such as for AlexNet [KSH12] and VGG-16 [SZ14].” §2.2.2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the combination to incorporate a ratio greater than or equal to 5 of Thoma.
Given the advantage of maximizing efficiency, reduce cost, and speed up processing, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claims 9, 13, 6 are rejected on the same grounds as claims 1, 5, 8 respectively.
Claim(s) 2-4, 10-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yong and Thoma, in view of Ding et al. (hereinafter Ding), Diverse Branch Block: Building a Convolution as an Inception-like Unit.
Regarding Claim 2, Yong and Thoma disclose the method as claimed in claim 1. Yong further discloses wherein optimizing the one or more convolution layers comprises: receiving an inference graph [“The model may be defined by nodes, branches between the nodes, at least one layer, functions of the at least one layer, and weight values of the branches.” ¶83; Fig. 6].
However, Yong fails to explicitly disclose computing a number of channels required for a preceding convolution layer based on the received graph, wherein the preceding convolution layer is preceding to the one or more convolution layers;
adding a number of filters in the preceding convolution layer by adding a plurality of dummy weights.
Thoma discloses computing a number of channels required for a preceding convolution layer based on the received graph, wherein the preceding convolution layer is preceding to the one or more convolution layers [“A linear image filter (also called a filter bank or a kernel) is an element F є Rkwxkhxd, where kw represents the filter’s width, kh the filter’s height and d the number of input channels. The filter F is convolved with the image I є Rwxhxd to produce a new image Il” ¶2.1 ¶1.];
adding a number of filters in the preceding convolution layer by adding a plurality of dummy weights [“adding more filters to that layer could improve the performance” §2.5.8 ¶2; “with zero padding at the borders” §2.5.8 ¶3].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the combination to incorporate the channel and filter usage of Thoma.
Given the advantage of determining channels and adding filters for convolution, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Yong fails to explicitly disclose combining the one or more convolution layers placed in the one or more parallel branches into one convolution layer based on the added number of filters in the preceding convolution layer.
Ding discloses combining the one or more convolution layers placed in the one or more parallel branches into one convolution layer based on the added number of filters in the preceding convolution layer [“Transform VI: a conv for multi-scale convolutions Considering a kh × kw (kh _ K, kw _ K) kernel is equivalent to a K × K kernel with some zero entries, we can transform a kh × kw kernel into K × K via zero-padding. Specifically, 1 × 1, 1 × K and K × 1 conv are particularly practical as they can be efficiently implemented. The input should be padded to align the sliding windows (Fig. 4).” §3.2 § Transform VI; Fig. 2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Ding before him before the effective filing date of the claimed invention, to modify the combination to incorporate the combining of convolution layers of Ding.
Given the advantage of consolidating the neural network to run on hardware constrained systems, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 3, Yong, Thoma, and Ding disclose the method as claimed in claim 2. Yong further discloses further comprising: modifying the inference graph based on the re-arranging [“The model may be defined by nodes, branches between the nodes, at least one layer, functions of the at least one layer, and weight values of the branches.” ¶83; Fig. 6].
However, Yong fails to explicitly disclose selecting a common kernel size for the combined convolution layer, wherein the kernel size is greater than or equal to kernel size of each convolution layer of the one or more convolutional layers;
computing a number of first filters required in each convolution layer of each of the one or more parallel branches;
computing a number of second filters required in the combined convolution layer, wherein the number of second filters is equal to a product of the number of first filters in each convolution layer and a number of the one or more parallel branches;
adjusting a plurality of weights for the first filters in the one or more parallel branches based on the number of extra filters added in the preceding convolution layer and the number of second filters required in the combined convolution layer;
re-arranging the plurality of adjusted weights for filters in the combined convolution layer.
Ding discloses selecting a common kernel size for the combined convolution layer, wherein the kernel size is greater than or equal to kernel size of each convolution layer of the one or more convolutional layers [“KxK” Figure 2];
computing a number of first filters required in each convolution layer of each of the one or more parallel branches [“Kx1” and “1xK” Figure 2; “The parameters of a conv layer with C input channels, D output channels and kernel size K × K reside in the conv kernel, which is a 4th-order tensor F є RD×C×K×K, and an optional bias b є RD. It takes a C-channel feature map I є RC×H×W as input and outputs a D-channel feature map O є RD×H′ ×W′ , where H′ and W′ are determined by K, padding and stride configurations.” §3.1];
computing a number of second filters required in the combined convolution layer, wherein the number of second filters is equal to a product of the number of first filters in each convolution layer and a number of the one or more parallel branches [“KxK” Figure 2; “The parameters of a conv layer with C input channels, D output channels and kernel size K × K reside in the conv kernel, which is a 4th-order tensor F є RD×C×K×K, and an optional bias b є RD. It takes a C-channel feature map I є RC×H×W as input and outputs a D-channel feature map O є RD×H′ ×W′ , where H′ and W′ are determined by K, padding and stride configurations.” §3.1];
adjusting a plurality of weights for the first filters in the one or more parallel branches based on the number of extra filters added in the preceding convolution layer and the number of second filters required in the combined convolution layer [“Kx1” and “1xK” Figure 2; “an optional bias b є RD” §3.1; “data-dependent kernel re-parameterization, as it generated the weights for multiple kernels of the same shape, then derived a kernel as the weighted sum of all such kernels to participate in the convolution” §2.4];
re-arranging the plurality of adjusted weights for filters in the combined convolution layer [“KxK” Figure 2; “an optional bias b є RD” §3.1; “data-dependent kernel re-parameterization, as it generated the weights for multiple kernels of the same shape, then derived a kernel as the weighted sum of all such kernels to participate in the convolution” §2.4].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Ding before him before the effective filing date of the claimed invention, to modify the combination to incorporate the combining of convolution layers of Ding.
Given the advantage of consolidating the neural network to run on hardware constrained systems, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 4, Yong, Thoma, and Ding disclose the method as claimed in claim 3. Yong further discloses wherein adjusting the plurality of weights comprises:
modifying the number of extra filters in the preceding convolution layer such that the number of extra filters is equal to a number of channels of the one or more convolution layers after concatenation, wherein the preceding convolution layer is preceding to the one or more convolution layers [“a convolutional layer, as shown in FIG. 2A, the left side is a group of input feature maps (total c,=4 feature maps), and the right side is a group of output feature maps ( c,+i =5 feature maps). Assuming that the width and height of the input feature map are W, and H,, respectively, and the width and height of the output feature map are W,+i and H,+i, respectively. The convolutional layer may contain 5 filters, which are also known as convolution kernels, and each filter may correspond to an output feature map and may contain 4 kernels (representing a two-dimensional kernel, i.e., a two-dimensional part of the convolution kernel). The width and height of a kernel are usually referred to as kernel size (k(i)wxk(i)h, for example lxl, 3x3, 5x5, etc.).” ¶104; Examiner Note: Filters/Kernels are based on the size of the feature maps];
modifying a number of other filters in the one or more convolution layers based on the modified number of extra filters in the preceding convolution layer [“a convolutional layer, as shown in FIG. 2A, the left side is a group of input feature maps (total c,=4 feature maps), and the right side is a group of output feature maps ( c,+i =5 feature maps). Assuming that the width and height of the input feature map are W, and H,, respectively, and the width and height of the output feature map are W,+i and H,+i, respectively. The convolutional layer may contain 5 filters, which are also known as convolution kernels, and each filter may correspond to an output feature map and may contain 4 kernels (representing a two-dimensional kernel, i.e., a two-dimensional part of the convolution kernel). The width and height of a kernel are usually referred to as kernel size (k(i)wxk(i)h, for example lxl, 3x3, 5x5, etc.).” ¶104; Examiner Note: Filters/Kernels are based on the size of the feature maps].
Claims 10-12 are rejected on the same grounds as claims 2-4 respectively.
Claim(s) 6, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yong and Thoma, in view of Liu et al. (hereinafter Liu), Sparse Convolutional Neural Networks, further in view of Mehta, PermNet: Permuted Convolutional Neural Network.
Regarding Claim 6, Yong and Thoma disclose the method as claimed in claim 1.
However, Yong fails to explicitly disclose wherein, based on the at least one of the resampling layers being followed by the convolution layer, performing the inference comprises:
modifying weights for the convolution layer;
interleaving a plurality of dummy weights with the modified weights in filters of the convolutional layer.
Thoma discloses wherein, based on the at least one of the resampling layers being followed by the convolution layer, performing the inference comprises:
modifying weights for the convolution layer [“Figures 5.8 to 5.10 show how the weights changed while training” §5.1.3 ¶3];
interleaving a plurality of dummy weights with the modified weights in filters of the convolutional layer [“with zero padding at the borders” §2.5.8 ¶3].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong and Thoma before him before the effective filing date of the claimed invention, to modify the combination to incorporate weight modification and dummy weights of Thoma.
Given the advantage of more accurate models, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Yong fails to explicitly disclose respacing the interleaved weights in filters of the convolutional layer.
Liu discloses respacing the interleaved weights in filters of the convolutional layer [“An example sparse matrix B. The shadowed squares represent non-zero elements and the blank squares represent zero elements” Fig 3(a)].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Liu before him before the effective filing date of the claimed invention, to modify the combination to incorporate a sparse matrix of Liu.
Given the advantage of faster processing, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Yong fails to explicitly disclose performing a dimension scaling operation on an inference graph.
Mehta discloses performing a dimension scaling operation on an inference graph [“Model scaling. (a) The baseline network to be scaled. (b-d) Networks with width, depth and input image resolution scaled respectively.” Fig. 1.1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, Liu, and Mehta before him before the effective filing date of the claimed invention, to modify the combination to incorporate the scaling choices of Mehta.
Given the advantage of decreasing or increasing network complexity, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 14 is rejected on the same grounds as claim 6.
Claim(s) 7, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yong and Thoma, in view of Du et al. (hereinafter Du), License plate super-resolution reconstruction based on improved ESPCN network.
Regarding Claim 7, Yong and Thoma disclose the method as claimed in claim 1.
However, Yong fails to explicitly disclose wherein the one or more resampling layers includes one or more depth to space layers or one or more transpose convolution layers.
Du discloses wherein the one or more resampling layers includes one or more depth to space layers or one or more transpose convolution layers [“convolutional layer and the depth to space layer” §II.B ¶1; Fig. 3].
It would have been obvious to one having ordinary skill in the art, having the teachings of Yong, Thoma, and Du before him before the effective filing date of the claimed invention, to modify the combination to incorporate the depth to space layer of Du.
Given the advantage of upsampling to reconstruct images with accurate boundaries, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 15 is rejected on the same grounds as claim 7.
Examiner’s Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification.
Conclusion
Any prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. Applicant is reminded that in amending in response to a rejection of claims, the patentable novelty must be clearly shown in view of the state of the art disclosed by the references cited and the objections made. Applicant must also show how the amendments avoid such references and objections. See 37 CFR §1.111(c). Additionally when amending, in their remarks Applicant should particularly cite to the supporting paragraphs in the original disclosure for the amendments.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.B./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148