DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The present application, 17858449, filed 07/06/2022 claims foreign priority to CN202111198116.7, filed 10/14/2021.
Claim Objections
Claims 1-9 are objected to under 37 C.F.R. 1.71(a) which requires “full, clear, concise, and exact terms” as to enable any person skilled in the art or science to which the invention or discovery appertains, or with which it is most nearly connected, to make and use the same. The following should be corrected.
A. In claim 1 lines 29-30, “the depth dimension operation criterion” should read “the depthwise dimension operation criterion” instead for consistency of claim terminologies. Claims 2-5 inherit the same deficiency as claim 1 by reason of dependence.
B. In claim 2 lines 15-16, “the operation criterion in the depth dimension” should read “the depthwise dimension operation criterion” instead for consistency of claim terminologies. Claims 3-5 inherit the same deficiency as claim 2 by reason of dependence.
C. In claim 4 line 4, “the depthwise convolution parameter partition” should read “the corresponding depthwise convolution parameter partition” instead for consistency of claim terminologies.
D. In claim 6 line 10, “the depthwise convolution parameter partition” should read “the corresponding depthwise convolution parameter partition” instead for consistency of claim terminologies. Claims 7-9 inherit the same deficiency as claim 6 by reason of dependence.
E. In claim 6 lines 16-17, “the pointwise convolution parameter partition” should read “the corresponding pointwise convolution parameter partition” instead for consistency of claim terminologies. Claims 7-9 inherit the same deficiency as claim 6 by reason of dependence.
E. In claim 7 lines 4-5, “the depthwise convolution parameter partition and the pointwise convolution parameter partition” should read “the corresponding depthwise convolution parameter partition and the corresponding pointwise convolution parameter partition” instead for consistency of claim terminologies.
Claim Interpretation
The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim. If the claimed invention requires the first condition to occur, then the broadest reasonable interpretation of the claim requires step A. If the claimed invention requires both the first and second conditions to occur, then the broadest reasonable interpretation of the claim requires both steps A and B. See MPEP 2111.04 II for more information.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-5 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “step S(B)” in line 31. There is insufficient antecedent basis for this limitation in the claim. For purposes of examination, this is interpreted as step (B) instead. Claims 2-5 inherit the same deficiency as claim 1 by reason of dependence.
Claim 3 recites “the operation data partitions” in line 4. There is insufficient antecedent basis for this limitation in the claim. For purposes of examination, this is interpreted as operation data partitions.
Claim 4 recites “the depthwise convolution operation” in lines 2-3. There is insufficient antecedent basis for this limitation in the claim. It is unclear whether this is supposed to refer to the depthwise weighting operation or to something else. For purposes of examination, this is interpreted as a depthwise convolution operation.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Under Step 1, claims 1-5 recite a series of steps and, therefore, is a process.
Under Step 2A prong 1, claim 1 recites
A convolution operation method, applied to an operation apparatus, comprising:
(A) configuring the operation apparatus to prompt the operation apparatus to access, according to a partition rule, operation data, a set of depthwise convolution parameters and a set of pointwise convolution parameters stored in an external memory;
(B) reading and storing an operation data partition from the external memory to an internal memory;
(C) reading and storing a corresponding depthwise convolution parameter partition of the set of depthwise convolution parameters from the external memory to the internal memory, and accordingly performing a depthwise weighting operation on the operation data partition by a convolution operation circuit to generate a depthwise weighted partition;
(D) performing a depthwise offset operation on the depthwise weighted partition by the convolution operation circuit to generate a depthwise convolution operation result partition;
(E) reading and storing a corresponding pointwise convolution parameter partition of the set of pointwise convolution parameters from the external memory to the internal memory, accordingly performing a pointwise weighting operation on the depthwise convolution operation result partition by the convolution operation circuit to generate a pointwise weighted partition, and performing an accumulation process in a depth dimension on the pointwise weighted partition to generate an output partition, wherein the accumulation process accumulates the pointwise weighted partition and a previous output partition when the previous output partition exists;
(F) when the output partition meets a depthwise dimension operation criterion, performing a pointwise offset operation on the output partition by the convolution operation circuit to generate a pointwise convolution operation result partition to be stored to the external memory; when the depth dimension operation criterion is not met, configuring the output partition to be the previous output partition, and performing step S(B) to step (F) on a next operation data partition; and
(G) performing step (B) to step (F) on the next operation data partition until the operation data is completely operated.
The above limitations of repeatedly performing a depthwise weighting operation, a depthwise offset operation, a pointwise weighting operation, an accumulation process, and a pointwise offset operation amounts to processing mathematical calculations and falls within the “Mathematical Concepts” and/or “Mental Processes” grouping of abstract ideas. The steps of “performing”, “performing”, “performing”, “performing”, “performing” and “performing” is a process that under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, other than reciting “a convolution operation circuit”, nothing in the claim element precludes the step from practically being performed in the human mind. For example, but for the “a convolution operation circuit” language, the claim encompasses manually multiplying performing a depthwise convolution operation using a three 7x7 input matrices with three corresponding 3x3 weight matrices to generate three weighted result matrices; adding an offset value to each element of the weighted result matrices to generate depthwise convolution result matrices; performing a pointwise convolution operation using the depthwise convolution result matrices and corresponding pointwise convolution weights to generate pointwise weighted result matrices, accumulating the elements of the pointwise weighted result matrices in the channel dimension, and adding an offset value to each element of the accumulated pointwise weighted results and repeating the same steps for a next set of input matrices using pen and paper. Accordingly, the claim is directed to recite an abstract idea.
Under step 2A prong 2, the claim recites the following additional elements: configuring the operation apparatus to prompt the operation apparatus to access, according to a partition rule, operation data, a set of depthwise convolution parameters and a set of pointwise convolution parameters stored in an external memory; reading and storing an operation data partition from the external memory to an internal memory; reading and storing a corresponding depthwise convolution parameter partition of the set of depthwise convolution parameters from the external memory to the internal memory; and reading and storing a corresponding pointwise convolution parameter partition of the set of pointwise convolution parameters from the external memory to the internal memory. However, the additional elements of “operation apparatus”, “convolution operation circuit”, “internal memory” and “external memory” are recited at a high-level of generality (i.e., as a generic operation apparatus comprising a generic convolution operation circuit for performing convolution operation and a generic memory for storing input and output data; and a generic memory for storing input and output data) such that they amount to no more than merely reciting the words “apply it” (or an equivalent) with the judicial exception or merely as tools to implement the abstract idea. The additional elements of “configuring the operation apparatus to prompt the operation apparatus to access, according to a partition rule, operation data, a set of depthwise convolution parameters and a set of pointwise convolution parameters stored in an external memory”, “reading and storing an operation data partition from the external memory to an internal memory”, “reading and storing a corresponding depthwise convolution parameter partition of the set of depthwise convolution parameters from the external memory to the internal memory”, and “reading and storing a corresponding pointwise convolution parameter partition of the set of pointwise convolution parameters from the external memory to the internal memory” are merely adding insignificant extra-solution activities, i.e. mere data gathering and are merely generally linking the use of abstract idea to a particular technological environment or field of use by limiting the data gathering step to a particular data source (external memory). See MPEP 2106.05(h) for more information. The additional elements do not, individually or in combination, integrate the exception into a practical application. Accordingly, the claim is not integrated into a practical application.
Under step 2B, claim 1 does not include additional elements that, individually or in combination, are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “operation apparatus”, “convolution operation circuit”, “internal memory” and “external memory” are recited at a high-level of generality (i.e., as a generic operation apparatus comprising a generic convolution operation circuit for performing convolution operation and a generic memory for storing input and output data; and a generic memory for storing input and output data) such that they amount to no more than merely reciting the words “apply it” (or an equivalent) with the judicial exception or merely as tools to implement the abstract idea. The additional elements of “configuring the operation apparatus to prompt the operation apparatus to access, according to a partition rule, operation data, a set of depthwise convolution parameters and a set of pointwise convolution parameters stored in an external memory”, “reading and storing an operation data partition from the external memory to an internal memory”, “reading and storing a corresponding depthwise convolution parameter partition of the set of depthwise convolution parameters from the external memory to the internal memory”, and “reading and storing a corresponding pointwise convolution parameter partition of the set of pointwise convolution parameters from the external memory to the internal memory” are merely adding insignificant extra-solution activities, i.e. mere data gathering and are merely generally linking the use of abstract idea to a particular technological environment or field of use by limiting the data gathering step to a particular data source (external memory). See MPEP 2106.05(h) for more information. See MPEP 2106.05(d)(II) which states that the courts have recognized computer functions such as “Receiving or transmitting data over a network”, “Storing and retrieving information in memory” and “Performing repetitive calculations” as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. The claim does not recite additional elements that alone or in combination amount to an inventive concept. Accordingly, the claim does not amount to significantly more than the abstract idea.
Under step 2A prong 1, claims 2-5 recite the same abstract idea as claim 1 by reason of dependence. Further, claim 2 recites further details regarding the inputs to the depthwise and pointwise convolution and the operation criterion; and claim 3 recites further details regarding the partitioning of the operation data which falls within the “Mathematical Concepts” and/or “Mental Processes” grouping of abstract ideas. In particular claims 2-3 do not include additional elements that would require further analysis under step 2A prong 2 and step 2B. Accordingly, the claims are directed to recite an abstract idea.
Under the Alice Framework step 2A prong 2, claim 4 recites the following additional elements: wherein the internal memory has a storage capacity corresponding to the depthwise convolution operation, at least stores the operation data partition, the depthwise convolution parameter partition and the depthwise convolution operation result partition, and stores the previous output partition when the operation data partition is generated by partitioning the operation data at least according to the depth dimension. Claim 5 recites the following additional elements: wherein the internal memory is a static random access memory (SRAM), the external memory is a dynamic random access memory (DRAM), and the internal memory and the external memory transmit data in between through a direct memory access (DMA) circuit. However, the additional elements of “an SRAM”, “a DRAM” and “a DMA circuit” in claim 5 are recited at a high-level of generality (i.e., as a generic memory devices for storing data; and as a generic DMA circuit for transferring data in between memory device) such that they amount to no more than mere instructions using a generic computer component or merely as tools to implement the abstract idea. The additional elements of “wherein the internal memory has a storage capacity corresponding to the depthwise convolution operation, at least stores the operation data partition, the depthwise convolution parameter partition and the depthwise convolution operation result partition, and stores the previous output partition when the operation data partition is generated by partitioning the operation data at least according to the depth dimension” in claim 4 is also recited at a high-level of generality (i.e., as a generic internal memory for storing internal data) as is merely adding an insignificant extra-solution activity. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See MPEP 2106.05(f) for more information. The additional elements do not, individually or in combination, integrate the exception into a practical application. Accordingly, the claims are not integrated into a practical application.
Under the Alice Framework step 2B, claims 4-5 do not include additional elements that, individually or in combination, are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “an SRAM”, “a DRAM” and “a DMA circuit” in claim 5 are recited at a high-level of generality (i.e., as a generic memory devices for storing data; and as a generic DMA circuit for transferring data in between memory device) such that they amount to no more than mere instructions using a generic computer component or merely as tools to implement the abstract idea. The additional elements of “wherein the internal memory has a storage capacity corresponding to the depthwise convolution operation, at least stores the operation data partition, the depthwise convolution parameter partition and the depthwise convolution operation result partition, and stores the previous output partition when the operation data partition is generated by partitioning the operation data at least according to the depth dimension” in claim 4 is also recited at a high-level of generality (i.e., as a generic internal memory for storing internal data) as is merely adding an insignificant extra-solution activity. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See MPEP 2106.05(f) for more information. See MPEP 2106.05(d)(II) which states that the courts have recognized computer functions such as “Receiving or transmitting data over a network” and “Storing and retrieving information in memory” as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. The claim does not recite additional elements that alone or in combination amount to an inventive concept. Accordingly, the claims do not amount to significantly more than the abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Komuravelli et al. (US 20210334072 A1), hereinafter Komuravelli, in view of Liu et al. (NPL – “An FPGA-Based CNN Accelerator Integrating Depthwise Separable Convolution”), hereinafter Liu.
Regarding claim 6, Komuravelli teaches a convolution operation method, applied to an operation apparatus, the operation apparatus comprising operation circuit (Komuravelli Figs. 1-12 and paragraphs [0026, 0041, 0061] an operation apparatus – system 100; convolution operation circuit –PEs); the method comprising:
(Komuravelli paragraphs [0041-0042] “In some embodiments, a convolution data matrix is C channels deep and there are 64 processing elements as part of the hardware system. The C channels are divided into groups of 64 channels. From each group of 64 channels, one channel is assigned to each of the 64 processing elements. In some embodiments, each processing element processes a single channel of its assigned channels before processing the next assigned channel. In various embodiments, the 64 processing elements each process an assigned channel in parallel. Although 64 processing elements are used in the example, a different number of processing elements can be appropriate as well”; paragraph [0027, 0068);
performing a depthwise convolution operation on the operation data partition and the depthwise convolution parameter partition by the convolution operation circuit to generate a depthwise convolution operation result partition (Komuravelli Figs. 2-12 and paragraph [0064] “At 705, depthwise convolution results are determined for a channel of the data matrix. For example, a dot product engine unit performs depthwise convolution operations using the depthwise convolution weights prepared at 701 and the corresponding convolution data prepared at 703 … The results of the depthwise convolution operation are depthwise convolution partial result elements or partial depthwise convolution results” ; operation data partition - corresponding convolution data prepared at 703; depthwise convolution parameter partition - depthwise convolution weights prepared at 701; depthwise convolution operation result partition - partial depthwise convolution results; paragraph [0045]);
performing a pointwise convolution operation on the depthwise convolution operation result partition and the pointwise convolution parameter partition by the convolution operation circuit to generate a pointwise convolution operation result partition (Komuravelli Figs. 2-12 and paragraph [0067, 0072-0073] “At 709, pointwise convolution results are determined. For example, the partial depthwise convolution results from step 705 along with corresponding pointwise convolution weights from step 707 are used as inputs to a pointwise convolution unit … At 713, upstream and local element-wise multiplication results are accumulated … the reduction unit sums the results corresponding to the different channels of the different pointwise convolution weight filters in parallel”; pointwise convolution parameter partition - corresponding pointwise convolution weights; pointwise convolution operation result partition - accumulated pointwise convolution results); and
storing the pointwise convolution operation result partition (Komuravelli paragraph [0074] “At 715, the final pointwise convolution results may be written to a memory location or another appropriate storage location”);
wherein, the depthwise convolution operation result partition is not stored to the external memory (Komuravelli Fig. 2 and paragraph [0064] “the output of the dot product engine is transmitted to and utilized as an input for step 709. For example, as depthwise convolution results are determined, the results are transmitted to a pointwise convolution unit, such as an element-wise multiplication unit, even as additional depthwise convolution results are being calculated”).
Komuravelli does not explicitly teach the operation apparatus comprising an internal memory, a convolution operation circuit and a direct memory access (DMA) circuit; storing an operation data partition of operation data and a corresponding depthwise convolution parameter partition in a set of depthwise convolution parameters from an external memory to the internal memory by the DMA circuit according to a partition rule; storing a corresponding pointwise convolution parameter partition in a set of pointwise convolution parameters from the external memory to the internal memory by the DMA circuit according to the partition rule; storing the pointwise convolution operation result partition to the external memory by the DMA circuit.
However, on the same field of endeavor, Liu discloses an operation apparatus comprising an internal memory, a convolution operation circuit and a direct memory access (DMA) circuit; storing an operation data and a corresponding convolution parameter from an external memory to the internal memory by the DMA circuit; and storing the convolution operation result to the external memory by the DMA circuit (Liu Figs. 3-4 and page 6-7 section 3.1-3.2; internal memory – on-chip buffers; direct memory access (DMA) circuit – DMA; external memory – DDR).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Komuravelli using Liu and configure the system 100 of Komuravelli to include an internal memory including an input/weight buffer and an output buffer; and a DMA circuit. Further, configure the system to read and store the input data channels and the corresponding depthwise and pointwise weight matrices from an external memory to the input/weight buffer and store and transmit the accumulated pointwise convolution results from the output buffer to the external memory in order to be able to process large input matrices by storing the input matrix and weights in the external memory and processing slices of the input matrix by loading and storing the slices to the on-chip buffer and processing the slices that are stored on-chip (Liu page 6 section 3.1 and page 8-9 section 3.2.3).
Therefore, the combination of Komuravelli as modified in view of Liu teaches a convolution operation method, applied to an operation apparatus, the operation apparatus comprising an internal memory, a convolution operation circuit and a direct memory access (DMA) circuit; the method comprising: storing an operation data partition of operation data and a corresponding depthwise convolution parameter partition in a set of depthwise convolution parameters from an external memory to the internal memory by the DMA circuit according to a partition rule; storing a corresponding pointwise convolution parameter partition in a set of pointwise convolution parameters from the external memory to the internal memory by the DMA circuit according to the partition rule; storing the pointwise convolution operation result partition to the external memory by the DMA circuit.
Regarding claim 9, Komuravelli as modified in view of Liu teaches all the limitations of claim 6 as stated above.
Komuravelli does not explicitly teach wherein the partition rule is determined by a storage capacity of the internal memory.
However, on the same field of endeavor, Liu discloses a partition rule that is based at least on the size of internal memory (Liu page 8 section 3.2.3 “Since the on-chip cache resources are often extremely limited, this is usually unsatisfactory for all input feature maps and weights to be cached on the chip. The data must be partitioned”).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Komuravelli and generalize the teaching of Liu and partition the input matrix and corresponding weights based at least on the size of the on-chip buffers such that the matrix partitioned matrix and weights will fit on the on-chip buffers in order to maximize bandwidth and reduce the delay caused by on-chip off-chip data exchange (Liu page 16 section 5).
Therefore, the combination of Komuravelli as modified in view of Liu teaches wherein the partition rule is determined by a storage capacity of the internal memory.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Komuravelli in view of Liu as applied to claim 6 above, and further in view of Li et al. (NPL – “Block Convolution: Toward Memory-Efficient Inference of Large-Scale CNNs on FPGA”), hereinafter Li, and Baddiley (US 4852065 A).
Regarding claim 7, Komuravelli as modified in view of Liu teaches all the limitations of claim 6 as stated above. Further, Komuravelli as modified in view of Liu teaches wherein the internal memory comprises a first area and a second area (Liu Fig. 3; first area – image/weight buffer; second area – output buffer).
Komuravelli does not explicitly teach the first area is time-division multiplexed for the operation data partition, the depthwise convolution parameter partition, the depthwise convolution operation result partition and the pointwise convolution parameter partition, and the second area is exclusive to output data of the pointwise convolution operation.
However, on the same field of endeavor, Li discloses a memory comprising a first and second area where the first area are used for storing convolution inputs as well as the intermediate results (Li Fig. 12 and page 1444 section III.B.3; first area – intermediate buffer).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Komuravelli in view of Liu and generalize the teaching of Li by configuring the image/weight buffer as an intermediate buffer that also stores intermediate calculation result. Doing so would result in the output buffer being exclusive for storing the final output result (pointwise convolution operation result). Komuravelli paragraphs [0041-0044] discloses that the pointwise convolution operation result may have a larger size than the input matrix partition i.e. 128 pointwise convolution operation result compared to 64 input matrix channel for one partition. Therefore, it would be obvious to make the output buffer exclusive for storing the pointwise convolution operation result such that its size does not have to be increased as compared to having other additional data being stored in the output buffer.
Therefore, the combination of Komuravelli as modified in view of Liu and Li teaches the second area is exclusive to output data of the pointwise convolution operation.
Komuravelli as modified in view of Liu and Li does not explicitly teach the first area is time-division multiplexed for the operation data partition, the depthwise convolution parameter partition, the depthwise convolution operation result partition and the pointwise convolution parameter partition.
However, on the same field of endeavor, Baddiley discloses a buffer in which its inputs and outputs are time-division multiplexed (Baddiley abstract).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Komuravelli in view of Liu and Li and generalize the teaching of Baddiley and implement time-division multiplexing on the inputs and outputs of the intermediate buffer in order to reduce the required width of the intermediate buffer (Baddiley abstract).
Therefore, the combination of Komuravelli as modified in view of Liu, Li and Baddiley teaches the first area is time-division multiplexed for the operation data partition, the depthwise convolution parameter partition, the depthwise convolution operation result partition and the pointwise convolution parameter partition.
Claim 8 is are rejected under 35 U.S.C. 103 as being unpatentable over Komuravelli in view of Liu as applied to claim 6 above, and further in view of Liu et al. (US 20210224125 A1), hereinafter L2.
Regarding claim 8, Komuravelli as modified in view of Liu teaches all the limitations of claim 6 as stated above.
Komuravelli does not explicitly teach wherein the operation apparatus further comprises a processing circuit, the method further comprising: configuring, according to the partition rule, the processing circuit to control the DMA circuit to read the operation data, the set of depthwise convolution parameters and the set of pointwise convolution parameters stored in the external memory.
However, on the same field, L2 discloses an operation circuit comprising a processing circuit configured to control data reading/data transfer process performed by a DMA controller (L2 Fig. 16 and paragraphs [0108-0109] “The direct memory access controller (DMAC) no is configured to transfer the input data or an input matrix in an external memory 40 to various memories (buffer), for example, transfer the weight data to the weight memory 106 or transfer the input data to the third memory (input memory) 104. Alternatively, the DMAC no transfers the output data from the unified memory 109 to a DDR/HBM. Optionally, a complete DMA transmission process includes four steps: a DMA request, a DMA response, DMA transmission, and DMA ending. The external memory 40 may be a double data rate synchronous dynamic random access memory (DDR), a high bandwidth memory (HBM), or the like. A control unit (Flow control) 111 is configured to control a processing process and control the data reading manner”; processing circuit – control unit 111).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Komuravelli using L2 and configure the system 100 to include a control unit for controlling the overall operation of the system including controlling the convolution processing and the data reading processing (L2 paragraph [0109]).
Therefore, the combination of Komuravelli as modified in view of Liu and L2 teaches wherein the operation apparatus further comprises a processing circuit, the method further comprising: configuring, according to the partition rule, the processing circuit to control the DMA circuit to read the operation data, the set of depthwise convolution parameters and the set of pointwise convolution parameters stored in the external memory.
Claims 1-4 are rejected under 35 U.S.C. 103 as being unpatentable over Komuravelli in view of Liu and Bouguezzi et al. (NPL – “An Efficient FPGA-Based Convolutional Neural Network for Classification: Ad-MobileNet”), hereinafter Bouguezzi.
Regarding claim 1, Komuravelli teaches an convolution operation method, applied to an operation apparatus, comprising (Komuravelli Figs. 1-12 and paragraphs [0026, 0041, 0061] operation apparatus – system 100):
(A) configuring the operation apparatus to prompt the operation apparatus to access, according to a partition rule, operation data, a set of depthwise convolution parameters and a set of pointwise convolution parameters (Komuravelli Figs. 2-12 and paragraph [0036] “At 301, a convolution operation is received. The convolution operation, such as a traditional convolution operation, includes operands such as a 3D convolution data matrix and convolution weight matrices … To improve the efficiency of the computation of the convolution operation, the convolution operation can be replaced with a combination of separable convolutions that approximate the convolution operation. In order to perform the separable convolutions more efficiently, the problem is decomposed into multiple smaller operations such as multiple partial depthwise and pointwise convolution operations”; paragraph [0049] “At 501, local instructions are received. For example, a depthwise convolution instruction, a pointwise convolution instruction, and/or a reduction instruction is received at a processing element … In various embodiments, the instruction includes a specific convolution operation and convolution arguments specifying the data elements to perform the convolution operation on”; the operations/instructions prompts the system to access the inputs/arguments for the operations/instructions);
(B) reading (Komuravelli paragraph [0050] “the local arguments corresponding to the local instruction are received. For example, data elements of a convolution data matrix along with corresponding data elements of depthwise convolution weight matrices are received”);
(C) reading (Komuravelli paragraphs [0050-0051] “the local arguments corresponding to the local instruction are received. For example, data elements of a convolution data matrix along with corresponding data elements of depthwise convolution weight matrices are received … At 503, local operations are performed. In some embodiments, a dot product engine performs depthwise convolution operations”);
(E) reading depthwise weighted partition (Komuravelli paragraphs [0050-0053] “the local arguments corresponding to the local instruction are received … As another example, the data elements may correspond to data elements of pointwise convolution weight filters … The depthwise convolution output results are transmitted to a local element-wise multiplication unit such as element-wise multiplication unit 217 of FIG. 2. Using an element-wise multiplication unit, a local pointwise convolution operation is performed by the processing element using output from the dot product engine … At 507, local and upstream convolution results are reduced. For example, a local partial pointwise convolution result computed by a local convolution operation performed at 503 is merged with an upstream partial pointwise convolution result received at 505”);
(F) when the depth dimension operation criterion is not met, configuring the output partition to be the previous output partition, and performing step (B) to step (F) on a next operation data partition (Komuravelli paragraph [0043, 0072-0074] “At 713, upstream and local element-wise multiplication results are accumulated … In the event additional downstream partial pointwise convolution results are needed to complete the pointwise convolution operation, the results are transmitted from the reduction unit to a downstream processing element via a reduction network using a point-to-point connection. For example, in some embodiments, the results are transmitted from a processing element such as processing element 211 of FIG. 2 to a downstream processing element such as processing element 221 of FIG. 2 via a point-to-point connection such as point-to-point connection 233 of FIG. 2. The transmitted results correspond to the running sum of partial pointwise convolution results for different pointwise convolution weight filters and will be summed together with corresponding downstream results … In some embodiments, the results accumulated at 713 are final pointwise convolution results. For example, the processing element is the final processing element in the reduction network chain for the pointwise convolution operation. In some embodiments, the processing element is the one assigned to process the last channel of the data matrix and the upstream accumulated result received at 711 corresponds to the accumulated results for all the other channels of the data matrix. At 715, the final pointwise convolution results may be written to a memory location or another appropriate storage location instead of a downstream processing element”; next operation data partition – channels 65-128; Note: step (F) includes two contingent limitations, however, both conditions cannot happen at the same time, therefore only one condition is required); and
(G) performing step (B) to step (F) on the next operation data partition until the operation data is completely operated (Komuravelli paragraph [0043] “For example, in one scenario, a convolution data matrix is C channels deep, there are C corresponding depthwise convolution weight matrices, and there are 64 processing elements as part of the hardware system. Each of the C depthwise convolution weight matrices is assigned to the processing element that is assigned its corresponding channel of the data matrix. For example, in the event C is 128, a first processing element can be assigned the 1st and 65th depthwise convolution weight matrix. A second processing element can be assigned the 2nd and 66th depthwise convolution weight matrix. The 64th processing element can be assigned the 64th and 128th depthwise convolution weight matrix. And so forth”; paragraph [0068] “For example, a 64-wide element-wise multiplication unit can compute 64 partial pointwise results in parallel. In the event there are 128 filters, two iterations are required to compute all 128 partial pointwise results”).
Komuravelli does not explicitly teach access, according to a partition rule, operation data, a set of depthwise convolution parameters and a set of pointwise convolution parameters stored in an external memory; reading and storing an operation data partition from the external memory to an internal memory; (C) reading and storing a corresponding depthwise convolution parameter partition of the set of depthwise convolution parameters from the external memory to the internal memory, D) performing a depthwise offset operation on the depthwise weighted partition by the convolution operation circuit to generate a depthwise convolution operation result partition; (E) reading and storing a corresponding pointwise convolution parameter partition of the set of pointwise convolution parameters from the external memory to the internal memory, accordingly performing a pointwise weighting operation on the depthwise convolution operation result partition by the convolution operation circuit to generate a pointwise weighted partition; and (F) when the output partition meets a depthwise dimension operation criterion, performing a pointwise offset operation on the output partition by the convolution operation circuit to generate a pointwise convolution operation result partition to be stored to the external memory.
However, on the same field of endeavor, Liu discloses an operation apparatus comprising an internal memory, a convolution operation circuit and a direct memory access (DMA) circuit; storing an operation data and a corresponding convolution parameter in an external memory; and reading and storing the operation data and the corresponding convolution parameter from the external memory to the internal using the DMA circuit; and storing the convolution operation result to the external memory using the DMA circuit (Liu Figs. 3-4 and page 6-7 section 3.1-3.2; internal memory – on-chip buffers; direct memory access (DMA) circuit – DMA; external memory – DDR).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Komuravelli using Liu and configure the system 100 of Komuravelli to include an internal memory including an input/weight buffer and an output buffer; and a DMA circuit. Further, configure the system to read and store the input data channels and the corresponding depthwise and pointwise weight matrices from an external memory to the input/weight buffer and store and transmit the accumulated pointwise convolution results from the output buffer to the external memory in order to be able to process large input matrices by storing the input matrix and weights in the external memory and processing slices of the input matrix by loading and storing the slices to the on-chip buffer and processing the slices that are stored on-chip (Liu page 6 section 3.1 and page 8-9 section 3.2.3).
Therefore, the combination of Komuravelli as modified in view of Liu teaches access, according to a partition rule, operation data, a set of depthwise convolution parameters and a set of pointwise convolution parameters stored in an external memory; reading and storing an operation data partition from the external memory to an internal memory; (C) reading and storing a corresponding depthwise convolution parameter partition of the set of depthwise convolution parameters from the external memory to the internal memory, (E) reading and storing a corresponding pointwise convolution parameter partition of the set of pointwise convolution parameters from the external memory to the internal memory, accordingly performing a pointwise weighting operation by the convolution operation circuit to generate a pointwise weighted partition.
Komuravelli as modified in view of Liu does not currently teach (D) performing a depthwise offset operation on the depthwise weighted partition by the convolution operation circuit to generate a depthwise convolution operation result partition; performing a pointwise weighting operation on the depthwise convolution operation result partition by the convolution operation circuit to generate a pointwise weighted partition; and (F) when the output partition meets a depthwise dimension operation criterion, performing a pointwise offset operation on the output partition by the convolution operation circuit to generate a pointwise convolution operation result partition to be stored to the external memory.
However, on the same field of endeavor, Bouguezzi discloses performing a depthwise offset operation after performing a depthwise convolution operation generate a depthwise convolution operation result and performing a pointwise offset operation after performing a pointwise convolution operation to generate a pointwise convolution operation result (Bouguezzi Figs. 13-14; depthwise offset operation – bias addition in the depthwise convolution unit; pointwise offset operation - bias addition in the pointwise convolution unit).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Komuravelli in view of Liu using Bouguezzi and configure the system to add a bias on the partial depthwise convolution results to generate a depthwise convolution operation result partition which are then passed and used as input to the pointwise convolutions, then a bias to the accumulated pointwise convolution results to generate a pointwise convolution operation result partition that is stored to the external memory because adding a bias to the sum of weighted inputs is a normal operation when implementing a convolutional layer of a neural network (Bouguezzi page 3 section 3.1, page 16 top).
Therefore, the combination of Komuravelli as modified in view of Liu and Bouguezzi teaches (D) performing a depthwise offset operation on the depthwise weighted partition by the convolution operation circuit to generate a depthwise convolution operation result partition; performing a pointwise weighting operation on the depthwise convolution operation result partition by the convolution operation circuit to generate a pointwise weighted partition; and (F) when the output partition meets a depthwise dimension operation criterion, performing a pointwise offset operation on the output partition by the convolution operation circuit to generate a pointwise convolution operation result partition to be stored to the external memory.
Regarding claim 2, Komuravelli as modified in view of Liu and Bouguezzi teaches all the limitations of claim 1 as stated above. Further, Komuravelli as modified in view of Liu and Bouguezzi teaches
wherein the set of depthwise convolution parameters includes a set of depthwise convolution weights and a set of depthwise convolution offsets, and the set of pointwise convolution parameters includes a set of pointwise convolution weights and a set of pointwise convolution offsets (Komuravelli paragraph [0036, 0044, 0066]; Bouguezzi Figs. 13-15);
each of the operation data, the set of depthwise convolution weights, the set of depthwise convolution offsets, the set of pointwise convolution weights, the set of pointwise convolution offsets has a width dimension, a height dimension and the depth dimension, and the set of pointwise convolution weights further includes a number dimension and corresponds to the depth dimension of the set of pointwise convolution offsets (Komuravelli paragraph [0036, 0044, 0066]; Bouguezzi Figs. 13-15; number dimension – K); and
when the operation data is not partitioned according to the depth dimension, or when the operation data is partitioned according to the depth dimension and the accumulation process in the depth dimension is completely performed for the output partition, the output partition is said to have met the operation criterion in the depth dimension (Komuravelli paragraph [0074] “the results accumulated at 713 are final pointwise convolution results. For example, the processing element is the final processing element in the reduction network chain for the pointwise convolution operation”; the operation criterion is met when the different pointwise convolution results in all channels have been accumulated).
Regarding claim 3, Komuravelli as modified in view of Liu and Bouguezzi