Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
The following action is in response to the communication(s) received on 12/15/2025.
As of the claims filed 12/15/2025:
Claims 1, 3, 4, and 20 have been amended.
Claims 1-20 are pending.
Claims 1 and 20 are independent claims.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/15/2025 has been entered.
Response to Arguments
Applicant’s arguments filed 12/15/2025 have been fully considered, but are not fully persuasive.
With respect to the rejection under 35 USC § 112:
The amendments to claims 3 and 4 have overcome the indefiniteness rejection. Thus, the indefiniteness rejection has been withdrawn.
With respect to the rejection under 35 USC § 101:
Applicant asserts that the limitation “executing, by the processor, the neural network and the optimized neural network using the given hardware configuration to generate respective performance metrics based on at least in part on hardware resource utilization” cannot be performed practically in the human mind, as the parameters, such as memory and bandwidth availability could vary in real time (p.8 last ¶). Examiner respectfully submits that such parameters of performance metrics are not recited in the claims. i.e., the performance metrics could be something generic such as runtime in seconds. Thus, the exact parameters cannot be read into the claim language, and generating performance metrics remain abstract ideas which could be performed in the human mind which are merely generically linked to a neural network executed on a processor.
Applicant further asserts that the human mind cannot perform “generating, by the processor, a composite kernel that performs the functions of the first and second kernel, based on the first and second kernel” and that this “ignores the specific computational requirements recited in the claims.” (p.9 ¶2). Examiner respectfully disagrees. The broadest reasonable interpretation of “a composite kernel” is merely a kernel that performs the functions of the first and second kernel, and a kernel is interpreted as a matrix of weights. The processor is merely applied to the abstract idea recited, in which a human mind can perform generating a composite kernel, especially since the claims do not recite any specific technological architecture of the kernel that makes it impractical to perform with aid of a pen and paper.
Applicant further asserts that Claim 1 requires that the neural network has a convolutional layer and a pooling layer, thus reciting a well-defined and identifiable part of a neural network architecture (p.10 ¶2). Examiner respectfully submits that, similar to above, the claims recite combining two kernels, or two matrices, to generate a composite kernel. The details regarding the layers are merely details to a model, which is an algorithm, rather than a technology, such as the method of training neural networks.
Applicant further asserts that the claimed invention recites improvements to a technology, such as “reduced storage, reduced system complexity”. As noted above, the improvement is not directed towards the storage technology or the technology of training a neural network, but rather a model, thus algorithm, which is an abstract idea.
Applicant further asserts that retrieval, execution, or storage are not generic and integrate the abstract idea into a practical application by facilitating the improved neural network model (p.11 ¶1). Examiner respectfully disagrees. The crux of the invention (i.e., where the claim is directed towards) appears to be the generation of the composite kernel and not the retrieval, execution, or storage of the composite kernel. Similarly, these limitations do not appear to improve a specific technology, rather just details in which generation of the composite kernels are being performed.
Applicant further asserts that generating the composite kernel is a specific, unconventional technical solution. (p.11, last ¶). Examiner respectfully submits that generating a combined kernel is the identified abstract idea and not an additional element (for the reasons given above). Thus, this limitation is not analyzed for reciting significantly more than the abstract idea.
With respect to the rejection under 35 USC § 102:
Applicant asserts that Lan does not teach generating a composite kernel which reduces the number of layers and thereby reduces the compute operations (p.12). Examiner respectfully submits that the claims do not specify such details regarding producing the composite kernel; additionally, the reduction of storage operations from Lan is merely one of the intended uses and does not preclude the art from producing a composite kernel via the corresponding feature maps generated from fusion in [0069].
Applicant further asserts that Lan does not teach a kernel which performs the function of two layers “in a single pass” (p.13 ¶1). Examiner respectfully disagrees. First, the claims do not recite that the composite kernel performs the function of two layers in a single pass. rather, it recites that the composite kernel performs the function of the first and second kernels. Second, Lan does teach that the feature maps via pyramid fusion generates a template fuse unit which is viewed as a new layer, which is interpreted as a “single pass” (Lan [0070] The pyramid fusion is usually a backward fusion based on a specific convolution layer… [0008] perform neural network computing according to the template fuse unit… [0071] The collection of these fused layers is called the template fuse unit and is viewed as a new layer or a self-defined layer.)
Applicant further asserts that the fusion in Lan is “merely a change in the storage of data” (p.14 ¶1). Examiner respectfully disagrees. Although fusion in Lan changes the storage of data onto the faster NRAM, the fused layers are performing the function of the first and second kernels for the reasons provided above. Thus, as currently recited, Lan remains teaching the claimed invention.
Applicant further asserts that Lan does not teach kernels which are combined (p.15 ¶1). Examiner respectfully submits that a kernel, under the broadest reasonable interpretation, corresponds to a matrix which could be used as a layer of a convolutional neural network. Lan clearly shows the combined kernels (layers) in [0069] and [0071].
Applicant further asserts that Lan does not teach replacing the first and second kernels with the composite kernels. This is unpersuasive, as the fusion is understood as replacing the functions of the first and second convolutional layers ([0069] [0071]). Lan also more explicitly teaches this limitation ([0201] …the computing apparatus 201 performs the aforementioned four self-defined layers to replace original 14 layers, thus achieving technical effects of reducing input/output overheads and improving resource benefits)
Applicant further asserts that Lan does not teach bypassing the strided pooling layer. This is unpersuasive, as [0070] (teaching that one of the layers could be a pooling layer); [0201] (showing that the fused layer is replacing, thus bypassing the original strided pooling layer the original layers) do teach this limitation.
Applicant further asserts:
PNG
media_image1.png
285
642
media_image1.png
Greyscale
Examiner respectfully submits that the argument has been considered, but is moot, since the comments apply to a mapping that is not used in the current rejection.
Lan teaches the composite kernel generation via [0070] “The pyramid fusion is usually a backward fusion based on a specific convolution layer and a specific pooling layer in the neural network…”; [0201] “…computing apparatus 201 performs the aforementioned four self-defined layers to replace original 14 layers…”, wherein the set of four self-defined layers replacing the original layers corresponds to the generated composite kernel.
The dependent claims remain rejected at least by virtue of dependency on the rejected parent claims.
With respect to the rejection under 35 USC § 103:
Applicant further asserts that Yu does not teach the amended limitations of claim 1. Examiner respectfully submits that, as described above, Lan does teach these limitations. Thus, the dependent claims remain rejected at least by virtue of dependency on the rejected parent claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites a… method, thus a process, one of the four statutory categories of patentable subject matter (Step 1). However, Claim 1 further recites:
Analysing… the neural network architecture information to identify convolutional layers in the neural network which have associated strided pooling layers; generating… a composite kernel, based on the first and second kernel, which is an evaluation or judgement that can be performed in the human mind;
generating… an optimized neural network, in which the first kernel for the convolutional layer and the second kernel for the strided pooling layer are replaced with the composite kernel for a combined convolutional and pooling layer, which is an evaluation or judgement that can be performed in the human mind;
to generate respective performance metrics…, which is an evaluation or judgement that can be performed in the human mind;
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
Computer-implemented…; by a processor; executing, by the processor, the neural network and the optimized neural network using the given hardware configuration…, as the performance of an abstract idea on a computer is not more than instructions to "apply it" on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application;
Retrieving… neural network architecture information for a neural network, the neural network architecture information comprising layer information and kernel information for the neural network; retrieving… a first kernel for a convolutional layer identified as having an associated strided pooling layer; retrieving… a second kernel for the strided pooling layer associated with the convolutional layer, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application;
based on the neural network and the composite kernel; based at least in part on hardware resource utilization, which merely specifies the particular field of use or particular technological environment in which the abstract idea is to be performed, which by MPEP 2106.05(h) cannot integrate the abstract idea into a practical application.
and storing, by the processor, based on the respective performance metrics, one of the neural network or the optimized neural network for later deployment, which is merely an insignificant extra-solution activity of data output, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application;
Thus, the claim is directed towards an abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because the particular field of use or particular technological environment (MPEP 2106.05(h)), implementation on a computer (MPEP 2106.05(f)), and the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 2, dependent upon Claim 1, further recites
calculating the size of the composite kernel; comparing the size of the composite kernel with the local memory size; and only generating the composite kernel if the composite kernel size is less than or equal to the local memory size., which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
retrieving local memory size information for a processing unit of the neural network, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 3, dependent upon Claim 2, further recites
calculating the total size of the composite kernel and the input feature map; comparing the total size of the composite kernel and the input feature map with the local memory size; and only generating the composite kernel if the total size of the composite kernel and the input feature map is less than or equal to the local memory size, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
retrieving the expected size of an input feature map to be provided to the convolutional layer, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application;
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 4, dependent upon Claim 3, further recites
calculating the total size of the composite kernel, the input feature map and the output feature map; comparing the total size of the composite kernel, the input feature map and the output feature map with the local memory size; and only generating the composite kernel if the total size of the composite kernel, the input feature map and the output feature map is less than or equal to the local memory size, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
retrieving the expected size of an output feature map produced by the convolutional layer, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application;
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 5, dependent upon Claim 2, further recites
calculating the total size of the composite kernel and the output feature map; comparing the total size of the composite kernel and the output feature map with the local memory size; and only generating the composite kernel if the total size of the composite kernel and the output feature map is less than or equal to the local memory size, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
retrieving the expected size of an output feature map produced by the convolutional layer, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 6, dependent upon Claim 1, further recites
determining the advantage of using the composite kernel over the use of the first and second kernel; and only generating the composite kernel if the advantage is greater than a predetermined threshold, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2 and 2B, the claim does not recite any new additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself. Thus, the claim is ineligible.
Claim 7, dependent upon Claim 1, further recites
the strided pooling layers are average pooling layers and/or strided depthwise separable layers, which are merely details of an abstract idea (generating a composite kernel).
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2 and 2B, the claim does not recite any new additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself. Thus, the claim is ineligible.
Claim 8, dependent upon Claim 1, further recites
the neural network is a convolutional neural network, which are merely details of an abstract idea (generating a composite kernel).
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2 and 2B, the claim does not recite any new additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself. Thus, the claim is ineligible.
Claim 9, dependent upon Claim 1, further recites
determining if either of the first or second kernel comprise accumulated values that are larger than a threshold value, wherein the threshold value is indicative of a value which will cause underflow or overflow, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2 and 2B, the claim does not recite any new additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself. Thus, the claim is ineligible.
Claim 10, dependent upon Claim 9, further recites
if one or more values are greater than the predetermined threshold, generating instructions, which is an evaluation or judgement that can be performed in the human mind;
to saturate said values when applying the composite kernel, which is a mathematical concept. (Note: the specification [0068] recites: (i.e. perform saturate arithmetic/accumulation); accumulation corresponds to a mathematical concept.)
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
and storing said instructions with the composite kernel, which is merely an insignificant extra-solution activity of data output, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 11, dependent upon Claim 9, further recites
if one or more values are greater than the predetermined threshold, generating instructions to switch to a larger input data type when applying the composite kernel, which is an evaluation or judgement that can be performed in the human mind;
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
storing said instructions with the composite kernel, which is merely an insignificant extra-solution activity of data output, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 12, dependent upon Claim 9, further recites
if one or more values are greater than the predetermined threshold, generating instructions to...; process an input feature map using the composite kernel's scaled values to produce an output feature map, which is an evaluation or judgement that can be performed in the human mind;
scale the values of the composite kernel by a factor; re-scale the output feature map with the factor, which is a mathematical concept.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
and storing said instructions with the composite kernel, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 13, dependent upon Claim 12, further recites
the factor is either predetermined or calculated based on the change in value required to avoid underflow or overflow, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2 and 2B, the claim does not recite any new additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself. Thus, the claim is ineligible.
Claim 14, dependent upon Claim 1, further recites
generating the composite kernel based on the first kernel, the second kernel and the one or more additional kernels, the composite kernel adapted to perform the functions of each kernel it is based on, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
retrieving one or more additional kernels for the convolutional layer, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). Thus, the claim is ineligible.
Claim 15, dependent upon Claim 1, further recites
identifying, based on the neural network architecture information, an activation function associated with the convolutional layer identified as having an associated strided pooling layer; and if said activation function is a non-identity activation function, determine the divergence between an output feature map produced by applying the composite kernel and an output feature map produced by applying the first and second kernel, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2 and 2B, the claim does not recite any new additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself. Thus, the claim is ineligible.
Claim 16, dependent upon Claim 15, further recites
if the determined divergence is greater than a predetermined divergence threshold, the composite kernel is discarded, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2 and 2B, the claim does not recite any new additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself. Thus, the claim is ineligible.
Claim 17, dependent upon Claim 15, further recites
if the determined divergence is greater than a predetermined divergence threshold, the method further comprises generating instructions... to reduce the divergence below the predetermined threshold, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
to retrain the neural network , as the performance of an abstract idea on a computer is not more than instructions to "apply it" on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more. Thus, the claim is ineligible.
Claim 18, dependent upon Claim 1, further recites no additional abstract ideas. However:
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored thereon, which when executed by at least one processor, cause the at least one processor to perform the steps of claim 1, as the performance of an abstract idea on a computer is not more than instructions to "apply it" on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application;
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more. Thus, the claim is ineligible.
Claim 19, dependent upon Claim 1, further recites no additional abstract ideas. However:
Under Step 2A Prong 2, the claim does not include any additional elements which integrate the abstract idea into a practical application, since the additional elements consist of:
A neural network driver comprising: a processor; and memory storing computer readable instructions which, when implemented by the processor, cause the processor to perform the steps of claim 1, as the performance of an abstract idea on a computer is not more than instructions to "apply it" on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards an abstract idea.
Further, under Step 2B, the additional element does not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more. Thus, the claim is ineligible.
Claim 20 recites A non-transitory computer-readable storage medium, thus an article of manufacture, one of the four statutory categories of patentable subject matter (Step 1). However, Claim 20 further recites:
perform a first operation on the received input feature map; ... perform a second operation on the received output; …performs both the first and second operation on the input feature map, which is a mathematical concept;
process the input feature map received by the convolutional layer…; thereby enabling the strided pooling layer to be bypassed, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.
Under Step 2A Prong 2, the claim recites:
comprising a set of computer-readable instructions stored thereon, which when executed by at least one processor, cause the at least one processor to implement a neural network, was the performance of an abstract idea on a computer is not more than instructions to 'apply it' on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application;
to be deployed based on an availability of hardware resources of a data processing system, was the performance of an abstract idea on a computer is not more than instructions to 'apply it' on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application;
receive an input feature map...; a strided pooling layer arranged to receive an output of the convolutional layer..., which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)); implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more. The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-14 and 18-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Lan et al., US 20230259746 A1 (hereinafter Lan)
Regarding Claim 1, Lan teaches:
A computer-implemented method (Lan [0011] A fourth aspect of the present disclosure discloses a computer readable storage medium, on which computer program codes for forward fusing a neural network are stored. When the computer program codes are run by a processing apparatus, the method is performed.)
for optimising machine learning processing for a given hardware configuration, the method comprising: retrieving, by the processor, neural network architecture information for a neural network, the neural network architecture information comprising layer information and kernel information for the neural network; (Lan [0069] If the first-layer convolution layer 810 and the second-layer convolution layer 811 are fused, which means to store the feature map 802 to the NRAM 431 (weights of the first-layer convolution layer 810 and the second-layer convolution layer 811 may also be stored in the WRAM 432), the number of accesses between the computing apparatus 201 and the DRAM 204 may be reduced, thereby improving execution efficiency of the whole neural network. Since the feature maps (such as the feature map 801, the feature map 802, and the feature map 803) involved in fusion look like an inverted pyramid in the context logic of the neural network model as a whole, the fusion is called a pyramid fusion.)
analysing, by the processor, the neural network architecture information to identify convolutional layers in the neural network which have associated strided pooling layers; (Lan [0038] A neural network is composed of an input layer, a convolution layer, an activation function, a pooling layer, and a fully connected layer, with several layers at least and hundreds of layers at most. [0147] a stride of the kernel of the convolution layer or the pooling layer…) (Note: Pooling layers which have respective strides correspond to strided pooling layers.)
retrieving, by the processor, a first kernel for a convolutional layer identified as having an associated strided pooling layer; retrieving, by the processor, a second kernel for the strided pooling layer associated with the convolutional layer; (Lan [0070] The pyramid fusion is usually a backward fusion based on a specific convolution layer and a specific pooling layer in the neural network. In other words, a starting layer of the fusion is the convolution layer or the pooling layer, and according to hardware conditions, the layer backward fuses a plurality of layers which may contain a plurality of convolution layers and a plurality of pooling layers.) (Note: the specific convolution layer corresponds to the first kernel; the specific pooling layer corresponds to the second kernel)
generating, by the processor, a composite kernel that performs the functions of the first and second kernel, based on the first and second kernel; (Lan [0070] The pyramid fusion is usually a backward fusion based on a specific convolution layer and a specific pooling layer in the neural network…
[0201] Going back to FIG. 16, in a step 1603, neural network computing is performed according to the template fuse unit. In the neural network 152, the computing apparatus 201 performs the neural network computing according to four self-defined layers composed of the template fusion unit 1501, the template fusion unit 1503, the template fusion unit 1504, and the template fusion unit 1505. In other words, when performing the neural network computing, the computing apparatus 201 performs the aforementioned four self-defined layers to replace original 14 layers, thus achieving technical effects of reducing input/output overheads and improving resource benefits) (Note: the four self-defined layers replacing the original layers corresponds to the composite kernel)
generating, by the processor, based on the neural network and the composite kernel, an optimized neural network, in which the first kernel for the convolutional layer and the second kernel for the strided pooling layer are replaced with the composite kernel for a combined convolutional and pooling layer; executing, by the processor, the neural network and the optimized neural network using the given hardware configuration…
(Lan [0070] The pyramid fusion is usually a backward fusion based on a specific convolution layer and a specific pooling layer in the neural network. In other words, a starting layer of the fusion is the convolution layer or the pooling layer, and according to hardware conditions, the layer backward fuses a plurality of layers which may contain a plurality of convolution layers and a plurality of pooling layers. [0008] A first aspect of the present disclosure discloses an integrated circuit apparatus for forward fusing a neural network, which includes a processing apparatus and a computing apparatus. The processing apparatus is configured to perform a fusion in a direction of a starting point of the neural network to create a template fuse unit. The computing apparatus is configured to perform neural network computing according to the template fuse unit… [0071] The collection of these fused layers is called the template fuse unit and is viewed as a new layer or a self-defined layer.
[0201] Going back to FIG. 16, in a step 1603, neural network computing is performed according to the template fuse unit. In the neural network 152, the computing apparatus 201 performs the neural network computing according to four self-defined layers composed of the template fusion unit 1501, the template fusion unit 1503, the template fusion unit 1504, and the template fusion unit 1505. In other words, when performing the neural network computing, the computing apparatus 201 performs the aforementioned four self-defined layers to replace original 14 layers, thus achieving technical effects of reducing input/output overheads and improving resource benefits.)
to generate respective performance metrics based at least in part on hardware resource utilization; (Lan [0069] If the first-layer convolution layer 810 and the second-layer convolution layer 811 are fused, which means to store the feature map 802 to the NRAM 431 (weights of the first-layer convolution layer 810 and the second-layer convolution layer 811 may also be stored in the WRAM 432), the number of accesses between the computing apparatus 201 and the DRAM 204 may be reduced, thereby improving execution efficiency of the whole neural network. Since the feature maps (such as the feature map 801, the feature map 802, and the feature map 803) involved in fusion look like an inverted pyramid in the context logic of the neural network model as a whole, the fusion is called a pyramid fusion.) (Note: the reduced number of accesses corresponds to the generated performance metrics)
and storing, by the processor, based on the respective performance metrics, one of the neural network or the optimized neural network for later deployment. (Lan [0071] Another embodiment of the present disclosure shows a new kind of fusion method. This kind of fusion method is implemented by using hardware structures of FIG. 1, FIG. 2, FIG. 3, and FIG. 4 described above. This kind of fusion is called a template fuse unit (TFU). The template fuse unit mainly fuses a plurality of layers into one layer flexibly through a certain fusion policy, so as to reduce input/output overheads of the network. The template fuse unit includes the pyramid fusion and other fusion methods described above. The collection of these fused layers is called the template fuse unit and is viewed as a new layer or a self-defined layer.
[0072] This embodiment loads a feature map and a weight required by the template fuse unit from the DRAM 204 to the SRAM 308 on the chip at a time.) (Note: loading the template fuse unit, which include the pyramid fusion, to the SRAM corresponds to storing the composite kernel)
Regarding Claim 2, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
The method of claim 1, further comprising: retrieving local memory size information for a processing unit of the neural network; calculating the size of the composite kernel; comparing the size of the composite kernel with the local memory size; and only generating the composite kernel if the composite kernel size is less than or equal to the local memory size. (Lan [0089] This embodiment loads a feature map and a weight required by the template fuse unit from the DRAM 204 to the SRAM 308 on the chip at a time. After the feature map is loaded into the SRAM 308, the feature map is called an on-chip unit map.) (Note: the on-chip unit map corresponds to the input feature map.
[0090] SRAM 308 is large enough to accommodate the whole feature map, the template fuse unit is the small map mode. It is required to be noted that in the large map mode, the on-chip unit map is just a part of the feature map, while in the small map mode, if the available space of the SRAM 308 is large enough, or the feature map is small enough, the SRAM 308 may be possible to accommodate a plurality of feature maps at a time. In other words, the on-chip unit map may include the plurality of feature maps.
[0146] Rule 13: W.sub.i+IN1+IN2≤S
[0147] In the small map mode, this rule is that the size of the space of the SRAM 308 is required to satisfy a following condition:
W.sub.i+IN1+IN2≤S.
[0148] In other words, a sum of storage space W.sub.i required by a weight of a sub-map i, storage space IN1 required by the on-chip unit map, and caching space IN2 is not larger than the available space of the SRAM 308. When the processing apparatus 203 judges that this rule is not satisfied, the processing apparatus 203 decreases the number of on-chip unit maps until this rule is satisfied.
) (Note: the weights of the sub-map i in the SRAM, which satisfies the small map mode rule, corresponds to the composite kernel)
Regarding Claim 3, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 2. Lan further teaches:
The method of claim 2, further comprising: retrieving the expected size of the input feature map to be provided to the convolutional layer; (Lan [0089] This embodiment loads a feature map and a weight required by the template fuse unit from the DRAM 204 to the SRAM 308 on the chip at a time. After the feature map is loaded into the SRAM 308, the feature map is called an on-chip unit map.) (Note: the on-chip unit map corresponds to the input feature map.)
calculating the total size of the composite kernel and the input feature map; comparing the total size of the composite kernel and the input feature map with the local memory size; and only generating the composite kernel if the total size of the composite kernel and the input feature map is less than or equal to the local memory size. (Lan [0090] SRAM 308 is large enough to accommodate the whole feature map, the template fuse unit is the small map mode. It is required to be noted that in the large map mode, the on-chip unit map is just a part of the feature map, while in the small map mode, if the available space of the SRAM 308 is large enough, or the feature map is small enough, the SRAM 308 may be possible to accommodate a plurality of feature maps at a time. In other words, the on-chip unit map may include the plurality of feature maps.
[0146] Rule 13: W.sub.i+IN1+IN2≤S
[0147] In the small map mode, this rule is that the size of the space of the SRAM 308 is required to satisfy a following condition:
W.sub.i+IN1+IN2≤S.
[0148] In other words, a sum of storage space W.sub.i required by a weight of a sub-map i, storage space IN1 required by the on-chip unit map, and caching space IN2 is not larger than the available space of the SRAM 308. When the processing apparatus 203 judges that this rule is not satisfied, the processing apparatus 203 decreases the number of on-chip unit maps until this rule is satisfied.
) (Note: W.sub.i corresponds to the composite kernel; the plurality of feature maps in the SRAM, which satisfies the small map mode, corresponds to the composite kernel)
Regarding Claim 4, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 3. Lan further teaches:
The method of claim 3, further comprising: retrieving the expected size of the output feature map produced by the convolutional layer; calculating the total size of the composite kernel, the input feature map and the output feature map; comparing the total size of the composite kernel, the input feature map and the output feature map with the local memory size; and only generating the composite kernel if the total size of the composite kernel, the input feature map and the output feature map is less than or equal to the local memory size. (Lan [0142] Assuming that a size of space of the SRAM 308 is S, storage space required by the on-chip unit map is IN, and storage space required by computing results of the on-chip unit map is OUT, then, this rule is that the size of the space of the SRAM 308 is required to satisfy following conditions.
[0152] Rule 15: SubOUTi+W.sub.i+1+IN2≤S
[0153] In the small map mode, this rule is that the size of the space of the SRAM 308 is required to satisfy a following condition:
SubOUTi+W.sub.i+1+IN2≤S.
[0154] In other words, a sum of storage space SubOUTi required by intermediate results of the sub-map i, storage space W.sub.i+1 required by a weight of a next sub-map, and the cache space IN2 is not larger than the available space of the SRAM 308. When the processing apparatus 203 judges that this rule is not satisfied, the processing apparatus 203 decreases the number of on-chip unit maps until this rule is satisfied.) (Note: W.sub.i corresponds to the composite kernel; IN2 corresponds to the input feature map; SubOUT1i corresponds to the output feature map.)
Regarding Claim 5, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 2. Lan further teaches:
The method of claim 2 further comprising: retrieving the expected size of an output feature map produced by the convolutional layer; calculating the total size of the composite kernel and the output feature map; comparing the total size of the composite kernel and the output feature map with the local memory size; and only generating the composite kernel if the total size of the composite kernel and the output feature map is less than or equal to the local memory size. (Lan [0142] Assuming that a size of space of the SRAM 308 is S, storage space required by the on-chip unit map is IN, and storage space required by computing results of the on-chip unit map is OUT, then, this rule is that the size of the space of the SRAM 308 is required to satisfy following conditions.
[0152] Rule 15: SubOUTi+W.sub.i+1+IN2≤S
[0153] In the small map mode, this rule is that the size of the space of the SRAM 308 is required to satisfy a following condition:
SubOUTi+W.sub.i+1+IN2≤S.
[0154] In other words, a sum of storage space SubOUTi required by intermediate results of the sub-map i, storage space W.sub.i+1 required by a weight of a next sub-map, and the cache space IN2 is not larger than the available space of the SRAM 308. When the processing apparatus 203 judges that this rule is not satisfied, the processing apparatus 203 decreases the number of on-chip unit maps until this rule is satisfied.) (Note: W.sub.i corresponds to the composite kernel; IN2 corresponds to the input feature map; SubOUT1i corresponds to the output feature map. SubOUTi + W.sub.i ≤ SubOUTi+W.sub.i+1+IN2≤S. Therefore, satisfying SubOUTi+W.sub.i+1+IN2≤S must satisfy SubOUTi + W.sub.i ≤ S.)
Regarding Claim 6, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
The method of claim 1, further comprising: determining the advantage of using the composite kernel over the use of the first and second kernel; and only generating the composite kernel if the advantage is greater than a predetermined threshold. (Lan [0089] This embodiment loads a feature map and a weight required by the template fuse unit from the DRAM 204 to the SRAM 308 on the chip at a time. After the feature map is loaded into the SRAM 308, the feature map is called an on-chip unit map.) (Note: the on-chip unit map corresponds to the input feature map.
[0090] SRAM 308 is large enough to accommodate the whole feature map, the template fuse unit is the small map mode. It is required to be noted that in the large map mode, the on-chip unit map is just a part of the feature map, while in the small map mode, if the available space of the SRAM 308 is large enough, or the feature map is small enough, the SRAM 308 may be possible to accommodate a plurality of feature maps at a time. In other words, the on-chip unit map may include the plurality of feature maps.
[0146] Rule 13: W.sub.i+IN1+IN2≤S
[0147] In the small map mode, this rule is that the size of the space of the SRAM 308 is required to satisfy a following condition:
W.sub.i+IN1+IN2≤S.
[0148] In other words, a sum of storage space W.sub.i required by a weight of a sub-map i, storage space IN1 required by the on-chip unit map, and caching space IN2 is not larger than the available space of the SRAM 308. When the processing apparatus 203 judges that this rule is not satisfied, the processing apparatus 203 decreases the number of on-chip unit maps until this rule is satisfied.
) (Note: W.sub.i+IN1+IN2 corresponds to the composite kernel; the plurality of feature maps in the SRAM, which satisfies the small map mode, corresponds to the composite kernel. Satisfying W.sub.i+IN1+IN2≤S corresponds to having a greater advantage than the predetermined threshold. Decreasing the number of on-chip units corresponds to not generating the composite kernel)
Regarding Claim 7, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
The method of claim 1, wherein the strided pooling layers are average pooling layers and/or strided depthwise separable layers. (Lan [0107] Here, the pooling includes various kinds of pooling, such as maximum pooling (maxpool) or average pooling (avgpool).)
Regarding Claim 8, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
The method of claim 1, wherein the neural network is a convolutional neural network. (Lan [0003] A neural network is composed of a plurality of neuron systems connected according to certain rules. Roughly, the neural network is composed of following four kinds of layers: an input layer, a convolution layer, a pooling layer, and a fully connected layer. [0004] The fully connected layer plays the role of a classifier in the whole convolution neural network…)
Regarding Claim 9, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
The method of claim 1, further comprising the steps of: determining if either of the first or second kernel comprise accumulated values that are larger than a threshold value, wherein the threshold value is indicative of a value which will cause underflow or overflow. (Lan [0095] In a step 1102, a fusion is performed based on the starting layer, and all rules of the fusion policy are checked one by one to create the template fuse unit. The processing apparatus 203 performs the fusion based on the starting layer and checks all rules of the fusion policy one by one, so as to create the template fuse unit. On the premise that all rules are satisfied, hardware resources of the computing apparatus 201 are sufficient to load data required for computing the template fuse unit at a time and then perform neural network computing according to the template fuse unit. In addition to the starting rule, the fusion policy may also include following rules for example.
[0073] More specifically, the SRAM 308 is one of important reference indexes of the fusion policy. A size of space of the SRAM 308 determines whether the template fuse unit is a large map mode or a small map mode. The small map mode and the large map mode refer to whether a feature map stored in the DRAM 204 may be moved to the SRAM 308 for processing at a time. The processing apparatus 203 compares storage space required by the feature map with available space of the SRAM 308. If the space of the SRAM 308 is insufficient to hold the feature map, the template fuse unit is the large map mode. If the space of the SRAM 308 is large enough to hold the whole feature map, the template fuse unit is the small map mode. It is required to be noted that in the large map mode, the on-chip unit map is just a part of the feature map, while in the small map mode, if the available space of the SRAM 308 is large enough, or the feature map is small enough, the SRAM 308 may be possible to hold a plurality of feature maps at a time. In other words, the on-chip unit map may include the plurality of feature maps.
) (Note: determining if the SRAM is insufficient to hold the feature map corresponds to an indication of an overflow.)
Regarding Claim 10, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 9. Lan further teaches:
The method of claim 9, further comprising: if one or more values are greater than the predetermined threshold, generating instructions to saturate said values when applying the composite kernel; and storing said instructions with the composite kernel. (Lan [0129] Rule 13: W.sub.1+IN1+IN2≤S
[0130] In the small map mode, this rule is that the size of the space of the SRAM 308 is required to satisfy a following condition:
W.sub.i+IN1+IN2≤S
[0131] In other words, a sum of storage space W.sub.i required by a weight of a sub-map i, storage space IN1 required by the on-chip unit map, and caching space IN2 is not larger than the available space of the SRAM 308. When the processing apparatus 203 judges that this rule is not satisfied, the processing apparatus 203 decreases the number of on-chip unit maps until this rule is satisfied.) (Note: decreasing the number of unit maps until the rule is satisfied corresponds to generating and storing instructions to saturate the values)
Regarding Claim 11, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 10. Lan further teaches:
The method of claim 9, further comprising: if one or more values are greater than the predetermined threshold, generating instructions to switch to a larger input data type when applying the composite kernel; and storing said instructions with the composite kernel. (Lan [0073] More specifically, the SRAM 308 is one of important reference indexes of the fusion policy. A size of space of the SRAM 308 determines whether the template fuse unit is a large map mode or a small map mode. The small map mode and the large map mode refer to whether a feature map stored in the DRAM 204 may be moved to the SRAM 308 for processing at a time. The processing apparatus 203 compares storage space required by the feature map with available space of the SRAM 308. If the space of the SRAM 308 is insufficient to hold the feature map, the template fuse unit is the large map mode.
[0074] If the template fuse unit is the large map mode, the feature map must be split before the feature map may be loaded into the computing apparatus 201. The processing apparatus 203 splits the feature map in the DRAM 204 until an on-chip unit map that is small enough to meet the space requirements of the SRAM 308 is generated, so that the on-chip unit map may be moved to the SRAM 308 for processing at a time. When the feature map is split, an input-dependent operation and an output-dependent operation may be generated.) (Note: the template fuse unit in the large map mode corresponds to switching to a larger data type; the instructions executed by the processing apparatus to split the feature map in the DRAM correspond to storing said instructions with the composite kernel.)
Regarding Claim 12, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 9. Lan further teaches:
The method of claim 9, further comprising: if one or more values are greater than the predetermined threshold, generating instructions to: scale the values of the composite kernel by a factor, process an input feature map using the composite kernel's scaled values to produce an output feature map; re-scale the output feature map with the factor; and storing said instructions with the composite kernel. (Lan [0080] In a step 1001, the processing apparatus 203 judges whether the storage space required by the feature map is larger than the available space of the SRAM 308. If the storage space required by the feature map is larger than the available space of the SRAM 308, it is represented that the feature map may not be loaded into the SRAM 308 at a time. Therefore, a step 1002 is performed to split the feature map. In this embodiment, the processing apparatus 203 preferentially chooses to split in the N dimension because no input-dependent operation or output-dependent operation will be generated. If splitting in the N dimension fails to meet the requirements, then, splitting in the H or W dimension is considered. At this time, the input-dependent operation or the output-dependent operation may be generated. This embodiment also supports splitting in the C dimension, especially splitting along a Cout direction. As such, one convolution is split into multiple convolution by means of data optimization, which allows the WRAM 432 to hold the weight. For example, the weight is split onto four processor cores 306. Therefore, as long as splitting in a certain dimension is processable by the computing apparatus 201, the splitting shall fall within the scope of the present disclosure.) (Note: splitting in the appropriate dimension corresponds to scaling the values of the composite kernel by a factor; the processing apparatus executing such splitting corresponds to the storage of said instructions with the composite kernel.)
Regarding Claim 13, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 12. Lan further teaches:
The method of claim 12, wherein the factor is either predetermined or calculated based on the change in value required to avoid underflow or overflow. (Lan [0080] In a step 1001, the processing apparatus 203 judges whether the storage space required by the feature map is larger than the available space of the SRAM 308. If the storage space required by the feature map is larger than the available space of the SRAM 308, it is represented that the feature map may not be loaded into the SRAM 308 at a time. Therefore, a step 1002 is performed to split the feature map.) (Note: ensuring the feature map is split to fit the SRAM’s available space corresponds to avoiding overflow.)
Regarding Claim 14, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
The method of claim 1, further comprising: retrieving one or more additional kernels for the convolutional layer; and generating the composite kernel based on the first kernel, the second kernel and the one or more additional kernels, the composite kernel adapted to perform the functions of each kernel it is based on. (Lan [0069] If the first-layer convolution layer 810 and the second-layer convolution layer 811 are fused, which means to store the feature map 802 to the NRAM 431 (weights of the first-layer convolution layer 810 and the second-layer convolution layer 811 may also be stored in the WRAM 432), the number of accesses between the computing apparatus 201 and the DRAM 204 may be reduced, thereby improving execution efficiency of the whole neural network. Since the feature maps (such as the feature map 801, the feature map 802, and the feature map 803) involved in fusion look like an inverted pyramid in the context logic of the neural network model as a whole, the fusion is called a pyramid fusion.) (Note: the fusion of the first and second convolution layers such that the weights are stored in the WRAM corresponds to the composite kernel adapted to perform the functions of each kernel.)
Regarding Claim 18, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored thereon, which when executed by at least one processor, cause the at least one processor to perform the steps of claim 1. (Lan [0048] FIG. 3 is a schematic diagram of an internal structure of the computing apparatus 201. The computing apparatus 201 is used for processing input data in computer vision, speech, natural language, and data mining. The computing apparatus 201 in the figure is designed in a multi-core hierarchical structure. The computing apparatus 201 serves as an on-chip system, which includes a plurality of clusters. Each cluster further includes a plurality of processor cores. In other words, the computing apparatus 201 is composed of SoC-cluster-processor core hierarchy.
[0053] The control unit 41 is used for coordinating and controlling work of the operation unit 42 and the storage unit 43, so as to complete a deep learning task. The control unit 41 includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used for acquiring an instruction from the processing apparatus 203. The instruction decode unit 412 is used for decoding the instruction acquired and sending a decoding result as control information to the operation unit 42 and the storage unit 43.)
Regarding Claim 19, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
A neural network driver comprising: a processor; and memory storing computer readable instructions which, when implemented by the processor, cause the processor to perform the steps of claim 1. (Lan [0048] FIG. 3 is a schematic diagram of an internal structure of the computing apparatus 201. The computing apparatus 201 is used for processing input data in computer vision, speech, natural language, and data mining. The computing apparatus 201 in the figure is designed in a multi-core hierarchical structure. The computing apparatus 201 serves as an on-chip system, which includes a plurality of clusters. Each cluster further includes a plurality of processor cores. In other words, the computing apparatus 201 is composed of SoC-cluster-processor core hierarchy.)
Regarding Claim 20, Lan teaches:
A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored thereon, which when executed by at least one processor, cause the at least one processor to implement… ([Lan [0011] A fourth aspect of the present disclosure discloses a computer readable storage medium, on which computer program codes for forward fusing a neural network are stored. When the computer program codes are run by a processing apparatus, the method is performed)
…a neural network comprising: a convolutional layer arranged to receive an input feature map and perform a first operation on the received input feature map; a strided pooling layer arranged to receive an output of the convolutional layer and perform a second operation on the received output; and a composite kernel…; and wherein , when the composite kernel is used to process the input feature map received by the convolutional layer, the composite kernel performs both the first and second operation on the input feature map,... (Lan [0068] If the layers are not fused, in performing computing of the first-layer convolution layer 810, the computing apparatus 201 reads the 5×5 feature sub-map 804 from the DRAM 204. After the computing, the computing apparatus 201 stores the 3×3 feature sub-map 805 back to the DRAM 204. Next, the computing apparatus 201 reads the 5×5 feature sub-map 806 from the DRAM 204. After the computing, the computing apparatus 201 stores the 3×3 feature sub-map 807 to the DRAM 204. In performing computing of the second-layer convolution layer 811, similarly, it is required to read the 3×3 feature sub-map 805 from the DRAM 204. After the computing, it is required to store the 1×1 feature sub-map 808 to the DRAM 204. Next, it is required to read the 3×3 feature sub-map 807 from the DRAM 204. After the computing, it is required to store the 1×1 feature sub-map 809 to the DRAM 204. It may be known from the above explanation that the feature map 802, as intermediate data, is repeatedly read and stored on the chip and off the chip, which extremely occupies system resources.
[0069] If the first-layer convolution layer 810 and the second-layer convolution layer 811 are fused, which means to store the feature map 802 to the NRAM 431 (weights of the first-layer convolution layer 810 and the second-layer convolution layer 811 may also be stored in the WRAM 432), the number of accesses between the computing apparatus 201 and the DRAM 204 may be reduced, thereby improving execution efficiency of the whole neural network. Since the feature maps (such as the feature map 801, the feature map 802, and the feature map 803) involved in fusion look like an inverted pyramid in the context logic of the neural network model as a whole, the fusion is called a pyramid fusion.
)
…thereby enabling the strided pooling layer to be bypassed (Lan [0070] The pyramid fusion is usually a backward fusion based on a specific convolution layer and a specific pooling layer in the neural network. In other words, a starting layer of the fusion is the convolution layer or the pooling layer… [0201] Going back to FIG. 16, in a step 1603, neural network computing is performed according to the template fuse unit. In the neural network 152, the computing apparatus 201 performs the neural network computing according to four self-defined layers composed of the template fusion unit 1501, the template fusion unit 1503, the template fusion unit 1504, and the template fusion unit 1505. In other words, when performing the neural network computing, the computing apparatus 201 performs the aforementioned four self-defined layers to replace original 14 layers, thus achieving technical effects of reducing input/output overheads and improving resource benefits.) (Note: fusing the layers replacing the original layers corresponds to enabling the (original) strided pooling layer to be bypassed)
Regarding the limitation to be deployed based on an availability of hardware resources of a data processing system configured to execute the neural network:
A claim containing a "recitation with respect to the manner in which a claimed apparatus is intended to be employed does not differentiate the claimed apparatus from a prior art apparatus" if the prior art apparatus teaches all the structural limitations of the claim (MPEP 2114 II).
Thus, Lan remains teaching the claimed limitation above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Lan further in view of Yu et al., US 11321604 B2 (hereinafter Yu)
Regarding Claim 15, Lan respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Lan further teaches:
The method of claim 1, further comprising: identifying, based on the neural network architecture information, an activation function associated with the convolutional layer identified as having an associated strided pooling layer; (Lan [0299] Article B31. The integrated circuit apparatus of article B24, where the template fuse unit includes a convolution or pooling layer; the fusion policy is that a sum of difference values between side lengths of a kernel of the convolution or pooling layer and a stride of the kernel of the convolution or pooling layer is not greater than a redundancy threshold; and when the processing apparatus judges that the fusion policy is not satisfied, the processing apparatus adjusts the template fuse unit until the fusion policy is satisfied.)
and if said activation function is a non-identity activation function, (Lan [0093] Taking an AlexNet neural network model in FIG. 6 as an example, the model has a total of 23 layers. Assuming that layers 1 to 5 are fused, when the starting rule means that the starting layer is the top unfused layer in the neural network, the processing apparatus 203 selects a ReLU activation layer of layer 6 as the starting layer and fuses backward (in other words, the processing apparatus 203 performs a fusion in a direction of layer 7).) (Note: the ReLU activation layer corresponds to a non-identity activation function)
Lan does not teach, but Yu further teaches:
determine the divergence between an output feature map produced by applying the composite kernel and an output feature map produced by applying the first and second kernel. (Yu [4th col 5th line] In some circumstances, neural network weight parameter pruning operations may be utilized to remove redundant connections within a neural network. As utilized herein, the term “redundant” in connection with a neural network weight parameter, connection, node, neuron, and/or the like refers to a weight parameter, connection, node, neuron, and/or the like that may be removed from a neural network model while maintaining a specified level of accuracy, such as after retraining operations) (Note: the level of accuracy corresponds to determining the divergence between the composite kernel and the first and second kernels separately.)
Yu and Lan are analogous to the present invention because both are from the same field of endeavor of optimizing convolutional neural networks. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of removing redundancies while maintaining accuracy from Yu into Lan’s method of implementing composite kernels. The motivation would be to “maintaining sufficient accuracy to at least significantly accomplish one or more specified objectives of a neural network and/or neural network model and/or the like” (Yu 6th col 6th line).
Regarding Claim 16, Lan/Yu respectively teaches and incorporates the claimed limitations and rejections of Claim 15. Yu further teaches:
The method of claim 15, wherein if the determined divergence is greater than a predetermined divergence threshold, the composite kernel is discarded. (Yu [4th col 5th line] In some circumstances, neural network weight parameter pruning operations may be utilized to remove redundant connections within a neural network. As utilized herein, the term “redundant” in connection with a neural network weight parameter, connection, node, neuron, and/or the like refers to a weight parameter, connection, node, neuron, and/or the like that may be removed from a neural network model while maintaining a specified level of accuracy, such as after retraining operations) (Note: removing the redundant connections corresponds to discarding the composite kernel that comprises a first and second kernel)
Regarding Claim 17, Lan/Yu respectively teaches and incorporates the claimed limitations and rejections of Claim 15. Yu further teaches:
The method of claim 15, wherein if the determined divergence is greater than a predetermined divergence threshold, the method further comprises generating instructions to retrain the neural network to reduce the divergence below the predetermined threshold. (Yu [10th col 10th line] In an embodiment, mask layers and/or masked nodes (e.g., nodes identified as redundant) may be removed, as depicted, for example, at block 424. As discussed above, nodes and/or feature maps having a corresponding Boolean parameter α of value “0” may be removed, in an embodiment. Further, in removing mask layers, output activations of remaining nodes may be propagated to a next layer, such as depicted in FIG. 3d , for example. Also, in an embodiment, a neural network model, such as DNN 410 after having undergone processing for compression (e.g., redundant node identification and/or removal), may be retrained, as depicted at block 425, yielding a compressed neural network model, such as compressed DNN 430.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEP HAN whose telephone number is (703)756-1346. The examiner can normally be reached Mon-Fri 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached on (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.H./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights