Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to the claims filed on 08/04/2023.
Claims 1-20 are presented for examination.
Information Disclosure Statement
The information disclosure statements (IDS) filed 08/04/2023; 04/30/2024 are in compliance with the provisions of 37 CFR 1.97 and 1.98. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
The following claimed benefit is acknowledged: the instant application, filed 08/04/2023 claims priority from foreign application KR10-2023-0029932, filed 03/07/2023.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35
U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Symes et al. (Patent. No. US 12361274– hereinafter, Symes).
Regrading claim 1, Symes teaches a processor-implemented method comprising: determining an operation sequence of a neural network based on dependency information of a tile constituting a feature map of the neural network and layer information of the neural network (Symes, [Col.3 ,lines 16-41], “In examples described herein, there is also provided a method comprising: receiving an instruction to perform a first operation on a first layer of a neural network; receiving block dependency data; receiving an instruction to perform a second operation on a second layer of the neural network; performing the first operation comprising dividing the first layer into a plurality of input blocks, and operating on the input blocks to generate a plurality of output blocks; and performing the second operation after the first operation has generated a set number of output blocks, the set number being defined by the block dependency data...” and [Col.4 ,lines 31-38], “For example, in the neural network of FIG. 2, the first feature map, which serves as the input layer to the first pooling operation, has the dimensions 55×55×96. Assuming each element of the input layer stores an 8-bit value, the size of the input layer is around 290 kB. By contrast, the local memory 33 of the NPU 30 may be of the order of 10 to 50 kB. An operation instruction may therefore additionally include a block size to be used by the NPU 30 when performing the operation.” Examiner’s note, determining the process of operating the neural network based on the command stream for instructing a first operation on the first layer of the neural network and second operation on the second layer of the neural network.);
and generating a first command for controlling a feature map memory (Symes, [Col.3, lines 27-35], “.. the command stream comprises an instruction to perform a first operation on a first layer of a neural network, block dependency data, and an instruction to perform a second operation on a second layer of the neural network; and in response to the command stream, the second processing unit:” and [Claim 1], “receive an instruction to perform a first operation of a neural network on a first input feature map stored in the local memory, the first input feature map serving as an input for the first operation of the neural network, and the first input feature map comprising a first multi-dimensional array of input values; receive block dependency data comprising a block dependency value representing a set number of output blocks to be generated by the first operation before performing a second operation” Examiner’s note, the command stream including the instruction to perform the first operation of generating the feature to store in the local memory, therefore, the first instruction is considered as the first command.)
and a second command for controlling an operator based on the operation sequence (Symes, (Col.1, lines 50-57]and an instruction to perform a second operation on a second layer of the neural network; and in response to the command stream, the second processing unit: performs the first operation comprising dividing the first layer into a plurality of input blocks, and operating on the input blocks to generate a plurality of output blocks; and performs the second operation after the first operation has generated a set number of output blocks, the set number being defined by the block dependency data.” Examiner’s note, the second instruction is considered as the second command to operate the second operation after the first operation.,
wherein the first command comprises information on a tile input to each of a plurality of memory queues constituting the feature map memory (Symes, [Col. 4, lines 23-28, lines 50-55, Fig. 3. ], “An instruction within the command stream may comprise the type of operation to be performed, the locations in the system memory 40 of the input layer, the output layer and, where applicable, the weights, along with other parameters relating to the operation, such as the number of kernels, kernel size, stride, padding and/or activation function.” And [Lines 50-55], “FIG. 3 shows an example of a convolution operation in which the input layer is divided into four input blocks. The height and depth of each input block is the same as that of the input layer, and the width of each input block is one quarter of the width of the input layer. After operating on each input block, the resulting output block may be written to the system memory 40 as a block of the output layer.” Examiner’s note, the instruction includes the type of the operation to be performed, and the locations in the system memory of the input layer and output layer, therefore, location of the output layer blocks in the system memory are considered as memory queues constituting the feature map memory , each input block is considered as the tile .) .
Regarding claim 2, Symes teaches the method of claim 1, wherein the dependency information of the tile comprises information on another tile that is used to perform an operation on a predetermined tile and comprises an overlap region (Symes, [Fig.4, Col.4, and Col.5, lines 1-10], “An instruction within the command stream may comprise the type of operation to be performed, the locations in the system memory 40 of the input layer, the output layer and, where applicable, the weights, along with other parameters relating to the operation, such as the number of kernels, kernel size, stride, padding and/or activation function.…FIG. 4 shows a further example of a convolution operation in which the input layer is again divided into four input blocks. The input layer and the convolution layer are unchanged from the example of FIG. 3. However, in this example, the width and height of each input block is the same as that of the input layer, and the depth of each input block is one quarter of the depth of the input layer. Since the convolution operation sums over all channels in the depth direction, the NPU 30 does not write the output layer to the system memory 40 until the operation on all four input blocks has been completed. The NPU 30 therefore operates on the first input block and stores the resulting output block, A0, to the local memory 33. After operating on a second input block, the NPU 30 adds the resulting output block, A1, to the first output block, A0. The NPU 30 then repeats this process for the third and fourth input blocks. After completing the operation on all four blocks, the NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, which in this instance happens to be the complete output layer.” Examiner’s note, the input layer is divided into four input blocks, wherein, each input block has the width and height as the same as that input layer and the depth is one quarter of the depth of the input layer that corresponds to the information of the tile comprises information on another tile. The Fig.4 show a generating process of the four input blocks, for example, the result of the A0 is added to the result of A1 and the process is repeated until completed of the all operation of the input blocks, then NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, therefore, the result of first input block is added to the result of second input block that corresponds to the perform an operation on a predetermined tile and comprises an overlap region.).
Regarding claim 3, Symes teaches the method of claim 1, wherein the second command comprises information on a tile comprised in a memory queue that is a target of an operation among the plurality of memory queues for the operator (Symes, [Col.1, lines 50-57] and an instruction to perform a second operation on a second layer of the neural network; and in response to the command stream.. and performs the second operation after the first operation has generated a set number of output blocks, the set number being defined by the block dependency data.” and [Fig.7, Col. 5, lines 54-67 and Col.6, lines 1-31], “FIG. 7 illustrates a further example of two consecutive operations. In this example, the first operation is again a convolution operation and the second operation is a pooling operation. When performing the first operation… which the NPU 30 then writes to the system memory as a second block of the output layer. It will therefore be appreciated that, in performing a particular operation of the neural network, the NPU 30 may operate on an input block more than once. When performing the second operation, the NPU 30 divides the input layer into two input blocks. The input blocks of the second operation are the same size as the output blocks of the first operation. Moreover, each input block of the second operation spans the entire width and height of the input layer. Accordingly, after the NPU 30 generates output block A3 and writes the accumulated block to the system memory, the NPU 30 is free to perform the second operation. The second operation is therefore free to generate output block B0 after the first operation generates output block A3. Likewise, the second operation is free to generate output block B1 after the first operation generates output block B7. There is again a block dependency between the two operations… The command stream may therefore include an instruction that defines the block dependency between two consecutive operations. More particularly, the instruction may comprise block dependency data, which the NPU 30 then uses in order to determine when to perform the second operation.” Examiner’s note, the second instruction to perform the second operation is after the first operation was completed and saved to the memory queues (output blocks) of the system memory, Fig.7 disclose the first operation and second operation of first input layer and second input layer, the second operation is performed after the completed of the first operation, each output result of each operation is save to the output block of the system memory, therefore the second instruction is considered as the second command comprises information on a tile comprised in a memory queue that is a target of an operation among the plurality of memory queues for the operator.).
Regarding claim 4, Symes teaches the method of claim 1, wherein the memory queues are separated and constituted based on a layer of the neural network (Symes, [Fig.7, Col. 5, lines 54-67 and Col.6, lines 1-31], “FIG. 7 illustrates a further example of two consecutive operations. In this example, the first operation is again a convolution operation and the second operation is a pooling operation. When performing the first operation, the NPU 30 again divides the input layer into four input blocks. However, on this occasion, there is insufficient local memory to store the input block, the output block and the relevant block of the convolutional layer, which comprises 64 kernels. The NPU 30 therefore operates on the first input block using the first 32 kernels (k1-k32) of the convolutional layer to generate output block A0. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A1, A2, A3. The NPU 30 adds the four output blocks together to generate an accumulated block, A0+A 1+A2+A3, which the NPU then writes to the system memory as a first block of the output layer. The NPU 30 then operates on the first input block using the second 32 kernels (k32-k64) of the convolutional layer to generate output block A4. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A5, A6, A7. The NPU 30 adds the four output blocks together to generate an accumulated block, A5+A6+A7+A8, which the NPU 30 then writes to the system memory as a second block of the output layer. It will therefore be appreciated that, in performing a particular operation of the neural network, the NPU 30 may operate on an input block more than once. When performing the second operation, the NPU 30 divides the input layer into two input blocks. The input blocks of the second operation are the same size as the output blocks of the first operation. Moreover, each input block of the second operation spans the entire width and height of the input layer. Accordingly, after the NPU 30 generates output block A3 and writes the accumulated block to the system memory, the NPU 30 is free to perform the second operation. The second operation is therefore free to generate output block B0 after the first operation generates output block A3. Likewise, the second operation is free to generate output block B1 after the first operation generates output block B7. There is again a block dependency between the two operations.” Examiner’s note, the second operation is performed after the four input blocks of the first operation are completed and save to the memory blocks of the system memory, each of the memory blocks in the system memory is separated and constituted based on the first input layer and second input layer, as it can be seen at the Fig.7 .).
Regarding claim 5, Symes teaches the method of claim 1, wherein each of the plurality of memory queues is constituted in a plurality of memory banks stored in a unit of tiles (Symes, [Col. 4, lines 23-29], “An instruction within the command stream may comprise the type of operation to be performed, the locations in the system memory 40 of the input layer, the output layer and, where applicable, the weights, along with other parameters relating to the operation, such as the number of kernels, kernel size, stride, padding and/or activation function.” And [Fig.7, Col. 5, lines 54-67 and Col.6, lines 1-31], “FIG. 7 illustrates a further example of two consecutive operations. In this example, the first operation is again a convolution operation and the second operation is a pooling operation. When performing the first operation, the NPU 30 again divides the input layer into four input blocks. However, on this occasion, there is insufficient local memory to store the input block, the output block and the relevant block of the convolutional layer, which comprises 64 kernels. The NPU 30 therefore operates on the first input block using the first 32 kernels (k1-k32) of the convolutional layer to generate output block A0. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A1, A2, A3. The NPU 30 adds the four output blocks together to generate an accumulated block, A0+A 1+A2+A3, which the NPU then writes to the system memory as a first block of the output layer. The NPU 30 then operates on the first input block using the second 32 kernels (k32-k64) of the convolutional layer to generate output block A4. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A5, A6, A7. The NPU 30 adds the four output blocks together to generate an accumulated block, A5+A6+A7+A8, which the NPU 30 then writes to the system memory as a second block of the output layer. It will therefore be appreciated that, in performing a particular operation of the neural network, the NPU 30 may operate on an input block more than once. When performing the second operation, the NPU 30 divides the input layer into two input blocks. The input blocks of the second operation are the same size as the output blocks of the first operation. Moreover, each input block of the second operation spans the entire width and height of the input layer. Accordingly, after the NPU 30 generates output block A3 and writes the accumulated block to the system memory, the NPU 30 is free to perform the second operation. The second operation is therefore free to generate output block B0 after the first operation generates output block A3. Likewise, the second operation is free to generate output block B1 after the first operation generates output block B7. There is again a block dependency between the two operations.” Examiner’s note, the second input layer of the second operation is performed after the first input layer are completed and save to the memory blocks of the system memory, therefore, the memory queue constituted the memory banks in the system memory (location where the input layer is stored), as it can be seen at the Fig.7.).
Regarding claim 6, Symes teaches the method of claim 1, wherein a tile constituting the feature map of the neural network is generated by dividing the feature map in a single direction (Symes, [Fig.4, Col. 4, lines 9-33], “n response to instructions within the command stream, the NPU 30 operates on an input layer and generates in response an output layer. The output layer then serves as the input layer for a subsequent operation of the neural network. The term ‘input layer’ should be understood to mean any data structure that serves as the input for an operation of the neural network. Similarly, the term ‘output layer’ should be understood to mean any data structure that is output by an operation of the neural network. Accordingly, the input layer and/or the output layer may a tensor of any rank. In the example of FIG. 2, the input data serves as the input layer for the first convolution operation. The resulting output layer is then a feature map, which subsequently serves as the input layer for the pooling operation.... For example, in the neural network of FIG. 2, the first feature map, which serves as the input layer to the first pooling operation, has the dimensions 55×55×96.” Examiner’s note, Fig.2 discloses the input layer is divided into the four input blocks and each of the feature map corresponds to each input block, wherein the feature map of the neural network is generating in single direction.).
Regarding claim 7, Symes teaches the method of claim 5, wherein, in response to a bit width in a first direction of the feature map being greater than a bit width of the memory bank, a bit width of the tile is determined based on the bit width of the memory bank (Symes, [Col. 4, lines 29-49], “The size of an input layer and/or output layer may exceed that of the local memory 33 of the NPU 30. For example, in the neural network of FIG. 2, the first feature map, which serves as the input layer to the first pooling operation, has the dimensions 55×55×96. Assuming each element of the input layer stores an 8-bit value, the size of the input layer is around 290 kB. By contrast, the local memory 33 of the NPU 30 may be of the order of 10 to 50 kB. An operation instruction may therefore additionally include a block size to be used by the NPU 30 when performing the operation. In response to an operation instruction that includes a block size, the NPU 30 divides the input layer into a plurality of input blocks defined by the block size. The NPU 30 then operates on each input block and generates an output block.” Examiner’s note, the size of the input layer is around 290 kB, but the local memory of the NPU 30 may be of the order of 10 to 50 kB, therefore, the input layer is divided into the four input blocks, each input/tile is determined based on the local memory block/memory banks store each input block.).
Regarding claim 9, Symes teaches a processor-implemented method comprising: obtaining information of a tile constituting a feature map of a neural network (Symes, [Col.3 ,lines 16-41], “In examples described herein, there is also provided a method comprising: receiving an instruction to perform a first operation on a first layer of a neural network; receiving block dependency data; receiving an instruction to perform a second operation on a second layer of the neural network; performing the first operation comprising dividing the first layer into a plurality of input blocks, and operating on the input blocks to generate a plurality of output blocks; and performing the second operation after the first operation has generated a set number of output blocks, the set number being defined by the block dependency data. In examples described herein, there is further provided a system comprising a first processing unit, and a second processing unit, wherein: the first processing unit outputs a command stream to the second processing unit; the command stream comprises an instruction to perform a first operation on a first layer of a neural network, block dependency data, and an instruction to perform a second operation on a second layer of the neural network; and in response to the command stream, the second processing unit: performs the first operation comprising dividing the first layer into a plurality of input blocks, and operating on the input blocks to generate a plurality of output blocks; and performs the second operation after the first operation has generated a set number of output blocks, the set number being defined by the block dependency data.” Examiner’s note, receiving the command stream including the first instruction to perform the first operation on the first layer of the neural network and the second instruction perform the second operation on the second layer of the neural network.);
determining an operation sequence of the neural network based on the information of the tile (Symes, [Fig.4, Col.4, and Col.5, lines 1-10], “An instruction within the command stream may comprise the type of operation to be performed, the locations in the system memory 40 of the input layer, the output layer and, where applicable, the weights, along with other parameters relating to the operation, such as the number of kernels, kernel size, stride, padding and/or activation function.…FIG. 4 shows a further example of a convolution operation in which the input layer is again divided into four input blocks. The input layer and the convolution layer are unchanged from the example of FIG. 3. However, in this example, the width and height of each input block is the same as that of the input layer, and the depth of each input block is one quarter of the depth of the input layer. Since the convolution operation sums over all channels in the depth direction, the NPU 30 does not write the output layer to the system memory 40 until the operation on all four input blocks has been completed. The NPU 30 therefore operates on the first input block and stores the resulting output block, A0, to the local memory 33. After operating on a second input block, the NPU 30 adds the resulting output block, A1, to the first output block, A0. The NPU 30 then repeats this process for the third and fourth input blocks. After completing the operation on all four blocks, the NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, which in this instance happens to be the complete output layer.” Examiner’s note, the input layer is divided into four input blocks, wherein, each input block has the width and height as the same as that input layer and the depth is one quarter of the depth of the input layer that corresponds to the information of the tile. The Fig.4 show a the four input blocks are operating in the sequence order, for example, the result of the A0 is added to the result of A1 and the process is repeated until completed of the all operation of the input blocks, then NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory as a block of the output layer,);
determining a first tile to be stored in a feature map memory based on the operation sequence (Symes, [Fig.3, Col.3, lines 50-57], “FIG. 3 shows an example of a convolution operation in which the input layer is divided into four input blocks. The height and depth of each input block is the same as that of the input layer, and the width of each input block is one quarter of the width of the input layer. After operating on each input block, the resulting output block may be written to the system memory 40 as a block of the output layer.” Examiner’s note, the result of each input block is written to each output block of the system memory, wherein, the operation of the A0-A3 are in sequence.).;
storing the first tile and a second tile that is used to perform an operation on the first tile and comprises an overlap region in a first memory queue of the feature map memory (Symes, [Fig.4, Col.4, and Col.5, lines 1-10], “An instruction within the command stream may comprise the type of operation to be performed, the locations in the system memory 40 of the input layer, the output layer and, where applicable, the weights, along with other parameters relating to the operation, such as the number of kernels, kernel size, stride, padding and/or activation function.…FIG. 4 shows a further example of a convolution operation in which the input layer is again divided into four input blocks. The input layer and the convolution layer are unchanged from the example of FIG. 3. However, in this example, the width and height of each input block is the same as that of the input layer, and the depth of each input block is one quarter of the depth of the input layer. Since the convolution operation sums over all channels in the depth direction, the NPU 30 does not write the output layer to the system memory 40 until the operation on all four input blocks has been completed. The NPU 30 therefore operates on the first input block and stores the resulting output block, A0, to the local memory 33. After operating on a second input block, the NPU 30 adds the resulting output block, A1, to the first output block, A0. The NPU 30 then repeats this process for the third and fourth input blocks. After completing the operation on all four blocks, the NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, which in this instance happens to be the complete output layer.” Examiner’s note, the input layer is divided into four input blocks, wherein, each input block has the width and height as the same as that input layer and the depth is one quarter of the depth of the input layer that corresponds to the information of the tile comprises information on another tile. The Fig.4 show a generating process of the four input blocks, for example, the result of the A0 is added to the result of A1 and the process is repeated until completed of the all operation of the input blocks, then NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, therefore, the result of all four blocks as accumulated block, A0+A1+A2+A3 is written to the system memory, that corresponds to the perform an operation on a predetermined tile and comprises an overlap region.)
performing an operation on the first tile based on the first tile and the second tile ((Symes, [Fig.4, Col.4, and Col.5, lines 1-10], FIG. 4 shows a further example of a convolution operation in which the input layer is again divided into four input blocks. The input layer and the convolution layer are unchanged from the example of FIG. 3. However, in this example, the width and height of each input block is the same as that of the input layer, and the depth of each input block is one quarter of the depth of the input layer. Since the convolution operation sums over all channels in the depth direction, the NPU 30 does not write the output layer to the system memory 40 until the operation on all four input blocks has been completed. The NPU 30 therefore operates on the first input block and stores the resulting output block, A0, to the local memory 33. After operating on a second input block, the NPU 30 adds the resulting output block, A1, to the first output block, A0. The NPU 30 then repeats this process for the third and fourth input blocks. After completing the operation on all four blocks, the NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, which in this instance happens to be the complete output layer.”
and storing an operation result in a second memory queue of the feature map memory (Symes, (Symes, [Fig.7, Col. 5, lines 54-67 and Col.6, lines 1-31], “FIG. 7 illustrates a further example of two consecutive operations. In this example, the first operation is again a convolution operation and the second operation is a pooling operation. When performing the first operation, the NPU 30 again divides the input layer into four input blocks. However, on this occasion, there is insufficient local memory to store the input block, the output block and the relevant block of the convolutional layer, which comprises 64 kernels. The NPU 30 therefore operates on the first input block using the first 32 kernels (k1-k32) of the convolutional layer to generate output block A0. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A1, A2, A3. The NPU 30 adds the four output blocks together to generate an accumulated block, A0+A 1+A2+A3, which the NPU then writes to the system memory as a first block of the output layer. The NPU 30 then operates on the first input block using the second 32 kernels (k32-k64) of the convolutional layer to generate output block A4. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A5, A6, A7. The NPU 30 adds the four output blocks together to generate an accumulated block, A5+A6+A7+A8, which the NPU 30 then writes to the system memory as a second block of the output layer.).
Regarding claim 10, Symes teaches the method of claim 9, wherein the determining of the operation sequence comprises determining the operation sequence based on dependency information of the tile (Symes, [Fig.7, Col. 5, lines 54-67 and Col.6, lines 1-31], “FIG. 7 illustrates a further example of two consecutive operations. In this example, the first operation is again a convolution operation and the second operation is a pooling operation. When performing the first operation, the NPU 30 again divides the input layer into four input blocks. However, on this occasion, there is insufficient local memory to store the input block, the output block and the relevant block of the convolutional layer, which comprises 64 kernels. The NPU 30 therefore operates on the first input block using the first 32 kernels (k1-k32) of the convolutional layer to generate output block A0. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A1, A2, A3. The NPU 30 adds the four output blocks together to generate an accumulated block, A0+A 1+A2+A3, which the NPU then writes to the system memory as a first block of the output layer. The NPU 30 then operates on the first input block using the second 32 kernels (k32-k64) of the convolutional layer to generate output block A4. The NPU 30 then repeats this process for the other three input blocks to generate output blocks A5, A6, A7. The NPU 30 adds the four output blocks together to generate an accumulated block, A5+A6+A7+A8, which the NPU 30 then writes to the system memory as a second block of the output layer. It will therefore be appreciated that, in performing a particular operation of the neural network, the NPU 30 may operate on an input block more than once. When performing the second operation, the NPU 30 divides the input layer into two input blocks. The input blocks of the second operation are the same size as the output blocks of the first operation. Moreover, each input block of the second operation spans the entire width and height of the input layer. Accordingly, after the NPU 30 generates output block A3 and writes the accumulated block to the system memory, the NPU 30 is free to perform the second operation. The second operation is therefore free to generate output block B0 after the first operation generates output block A3. Likewise, the second operation is free to generate output block B1 after the first operation generates output block B7. There is again a block dependency between the two operations.”).
Regarding claim 11, Symes teaches the method of claim 9, wherein the information of the tile is determined based on a shared boundary area of a tile constituting the feature map (Symes, [Fig.4, Col.4, and Col.5, lines 1-10], “An instruction within the command stream may comprise the type of operation to be performed, the locations in the system memory 40 of the input layer, the output layer and, where applicable, the weights, along with other parameters relating to the operation, such as the number of kernels, kernel size, stride, padding and/or activation function.…FIG. 4 shows a further example of a convolution operation in which the input layer is again divided into four input blocks. The input layer and the convolution layer are unchanged from the example of FIG. 3. However, in this example, the width and height of each input block is the same as that of the input layer, and the depth of each input block is one quarter of the depth of the input layer. Since the convolution operation sums over all channels in the depth direction, the NPU 30 does not write the output layer to the system memory 40 until the operation on all four input blocks has been completed. The NPU 30 therefore operates on the first input block and stores the resulting output block, A0, to the local memory 33. After operating on a second input block, the NPU 30 adds the resulting output block, A1, to the first output block, A0. The NPU 30 then repeats this process for the third and fourth input blocks. After completing the operation on all four blocks, the NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, which in this instance happens to be the complete output layer.” Examiner’s note, the input layer is divided into four input blocks, wherein, each input block has the width and height as the same as that input layer and the depth is one quarter of the depth of the input layer that corresponds to the information of the tile comprises information on another tile. The Fig.4 show a generating process of the four input blocks, for example, the result of the A0 is added to the result of A1 and the process is repeated until completed of the all operation of the input blocks, then NPU 30 writes the accumulated block, A0+A1+A2+A3, to the system memory 40 as a block of the output layer, therefore, the result of first input block is added to the result of second input block that corresponds to the information of the tile is determined based on a shared boundary area of a tile constituting the feature map.).
Regarding claim 13 is rejected for the same reason as the claim 1, since these claims recite the same limitation.
Regarding claim 14 is rejected for the same reason as the claim 2, since these claims recite the same limitation.
Regarding claim 15 is rejected for the same reason as the claim 3, since these claims recite the same limitation.
Regarding claim 16, Symes teaches the apparatus of claim 15, wherein the operator is configured to generate a second tile by processing the first tile based on the second command (Symes, [Col.1, lines 50-57] and an instruction to perform a second operation on a second layer of the neural network; and in response to the command stream.. and performs the second operation after the first operation has generated a set number of output blocks, the set number being defined by the block dependency data.” and [Fig.7, Col. 5, lines 54-67 and Col.6, lines 1-31], “FIG. 7 illustrates a further example of two consecutive operations. In this example, the first operation is again a convolution operation and the second operation is a pooling operation. When performing the first operation… which the NPU 30 then writes to the system memory as a second block of the output layer. It will therefore be appreciated that, in performing a particular operation of the neural network, the NPU 30 may operate on an input block more than once. When performing the second operation, the NPU 30 divides the input layer into two input blocks. The input blocks of the second operation are the same size as the output blocks of the first operation. Moreover, each input block of the second operation spans the entire width and height of the input layer. Accordingly, after the NPU 30 generates output block A3 and writes the accumulated block to the system memory, the NPU 30 is free to perform the second operation. The second operation is therefore free to generate output block B0 after the first operation generates output block A3. Likewise, the second operation is free to generate output block B1 after the first operation generates output block B7. There is again a block dependency between the two operations… The command stream may therefore include an instruction that defines the block dependency between two consecutive operations. More particularly, the instruction may comprise block dependency data, which the NPU 30 then uses in order to determine when to perform the second operation.” Examiner’s note, the second instruction to perform the second operation is after the first operation was completed and saved to the memory queues (output blocks) of the system memory, Fig.7 disclose the first operation and second operation of first input layer and second input layer, the second operation is performed after the completed of the first operation, each output result of each operation is save to the output block of the system memory, therefore the second tile is generated based on the first tile and the second command).
Regarding claim 17 is rejected for the same reason as the claim 4, since these claims recite the same limitation.
Regarding claim 18 is rejected for the same reason as the claim 5, since these claims recite the same limitation.
Regarding claim 19 is rejected for the same reason as the claim 6, since these claims recite the same limitation.
Regarding claim 20 is rejected for the same reason as the claim 7, since these claims recite the same limitation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 8, 12 are rejected under 35 U.S.C. 103 as being unpatentable over Symes et al. (Patent. No. US 12361274– hereinafter, Symes) in view of GUO et al. (PUB. NO. US20200293866-hereinafter, GUO).
Regarding claim 8, Symes teaches the method of the claim 1 but it does not teach teaches wherein the feature map memory comprises static random access memory (SRAM),
On the other hand, GUO teaches wherein the feature map memory comprises static random access memory (SRAM) (GUO, [Par.0019], “In one embodiment, the integrated circuit includes a static random access memory (SRAM) that stores input feature maps, output feature maps, and weights for a current layer of the neutral network model. A current layer is a layer that is currently being processed by the AI engine. Slices of an input feature map for the current layer can be chained with a first head to create a first linked list. After a slice of the input feature map has been processed by the AI engine, the input feature map slice can be removed from the first linked list. A corresponding output feature map slice can be generated by the AI engine after it completes processing the input feature map slice. The corresponding output feature map slice can be chained to a second head to create a second linked list for the current layer. When a last slice of the input feature map for the current layer is processed, the first head can be swapped with the second head so that output feature map slices for that layer in the second linked list can be provided as input feature map slices for a next layer of the neural network model.”
Symes and Guo are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the feature map memory, as taught by Symes, to include the feature map memory comprises static random access memory (SRAM), as taught by Guo. The modification would have been obvious because one of the ordinary skills in art would be motivated to increase the scalability and efficiency of the AI engine, (GUO, [Par.0020], “In various embodiments, linked lists or other chained data structures used to track feature map slices do not require the chained feature map slices to be adjacent to each other, thereby enabling the AI engine to better utilize the SRAM without physically partitioning the SRAM for storing input feature maps, output feature maps and weights maps for the current layer. The use of an O (1) scheduler map to schedule the computation with idle MAC units can improve utilization of MAC units the AI engine, thereby increasing the scalability and efficiency of the AI engine.”).
Regarding claim 12, Symes as modified in view of Guo teaches a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1, (GUO, [Par.0071], “Computer-readable storage medium 1509 may also be used to store the some software functionalities described above persistently. While computer-readable storage medium 1509 is shown in an exemplary embodiment to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The terms “computer-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media, or any other non-transitory machine-readable medium.”).
Symes and Guo are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the processor to perform the method the claim 1, as taught by Symes, to include the A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, as taught by Guo. The modification would have been obvious because one of the ordinary skills in art would be motivated to capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure, (GUO, [Par.0071], “Computer-readable storage medium 1509 may also be used to store the some software functionalities described above persistently. While computer-readable storage medium 1509 is shown in an exemplary embodiment to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The terms “computer-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media, or any other non-transitory machine-readable medium.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EM N TRIEU whose telephone number is (571)272-5747. The examiner can normally be reached on Mon-Fri from 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached on (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.T./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights