Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or
composition of matter, or any new and useful improvement thereof, may obtain a patent
therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an
abstract idea without significantly more.
1. A data processing method, comprising:
determining a target processing order for processing a plurality of data blocks in data, based on maximum storage capacity of a buffer,
wherein the data is divided into the plurality of data blocks for separate processing, and adjacent data blocks in the plurality of data blocks have overlapping data;
and sequentially processing the plurality of data blocks (…) based on the target processing order,
using a neural network
wherein at least a part of a processing result of a later-processed data block of the plurality of data blocks is a multiplexing of a processing result corresponding to the overlapping data between the later-processed data block and one or more earlier-processed data blocks of the plurality of data blocks, adjacent to the later-processed data block.
Claim 1 Step 1:
Claim 1 is directed to a data processing method, the method comprising: a series of steps, and is therefore directed to a process, which is one of the four statutory categories.
Claim 1 Step 2A, Prong One:
Limitation(s) a,c can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
Claim 1 Step 2A, Prong Two:
Limitation(s) 1 d provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer, which is not indicative of integration into a practical application. See MPEP 2106.04(d) and 2106.0S(f).
Limitation(s) 1 b,e amount(s) to insignificant extra-solution activity of necessary data gathering/outputting, which is not indicative of integration into a practical application. See MPEP 2106.04(d) and 2106.0S(g).
Claim 1Step 2B:
The combination of these elements amount to a method comprising steps which can be performed mentally on a computer and comprising insignificant extra-solution and well-understood, routine activity. Therefore, these elements, when considered individually and in combination, fail to add an inventive concept to the claim. Consequently, claim 1 as a whole does not amount to significantly more than the recited judicial exceptions and the claim is not eligible.
The following claim inherits from claim 1 and therefore inherits its rejections.
2. The method according to claim 1,
wherein the neural network includes one or more network layers,
and determining the target processing order for processing the plurality of data blocks in the data, based on the maximum storage capacity of the buffer, includes: determining a multiplexable data amount corresponding to a process result multiplexed by a data block and an adjacent data block of the data block, in the plurality of data blocks, wherein the multiplexable data amount represents a sum of data amounts corresponding to the processing result of the overlapping data corresponding to the data block and the adjacent data block at each of the one or more network layers;
dividing the buffer into one or more multiplexable data buffer areas, based on the maximum storage capacity and the multiplexable data amount;
and determining the target processing order for processing the plurality of data blocks in the data, based on a quantity of the one or more multiplexable data buffer areas.
Limitation(s) 2 b-d can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
Limitation(s) 2 a provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 2 and therefore inherits its rejections.
3. The method according to claim 2, wherein:
the adjacent data blocks include: data blocks of the plurality of data blocks, adjacent along a first direction, and data blocks of the plurality of data blocks, adjacent along a second direction, wherein the first direction is perpendicular to the second direction; and the multiplexable data amount includes: a first multiplexable data amount between the data blocks adjacent along the first direction, and a second multiplexable data amount between the data blocks adjacent along the second direction.
Limitation(s) 3 amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 2 and therefore inherits its rejections.
4. The method according to claim 3,
wherein dividing the buffer into the one or more multiplexable data buffer areas, based on the maximum storage capacity and the multiplexable data amount, includes: dividing an on-chip buffer of the buffer into one or more first-type multiplexable data buffer areas and one or more second-type multiplexable data buffer areas, based on the maximum storage capacity, the first multiplexable data amount and the second multiplexable data amount, wherein:
the one or more first-type multiplexable data buffers are configured to store a processing result corresponding to an overlapping data between two data blocks of the plurality of data blocks, adjacent along the first direction; and the one or more second-type multiplexable data buffer areas are configured to store a processing result corresponding to an overlapping data between two data blocks of the plurality of data blocks, adjacent along the second direction.
Limitation(s) 4 a can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
Limitation(s) 4 b amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 2 and therefore inherits its rejections.
5. The method according to claim 2,
wherein the neural network includes a convolutional neural network,
and determining the multiplexable data amount corresponding to the process result multiplexed by the data block and the adjacent data block of the data block, in the plurality of data blocks, includes: determining the multiplexable data amount, based on a quantity of convolutional layers of the convolutional neural network, a size of a convolutional kernel, and a preset output data size corresponding to the data,
wherein the preset output data size is characterized by a data size corresponding to a final processing result obtained through neural network processing based on a preset data size.
Limitation(s) 5 b can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
Limitation(s) 5 a provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer.
Limitation(s) 5 c amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 2 and therefore inherits its rejections.
6. The method according to claim 2,
wherein determining the target processing order for processing the plurality of data blocks in the data, based on the quantity of the one or more multiplexable data buffer areas, includes: determining a data multiplexing rate corresponding to each processing order of processing the plurality of data blocks based on the quantity of the one or more multiplexable data buffer areas, wherein the data multiplexing rate represents a multiplexing ratio of the processing result of the overlapping data in the plurality of data blocks;
and determining the target processing order based on the data multiplexing rate corresponding to each processing order.
Limitation(s) 6 a,b can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 1 and therefore inherits its rejections.
7. The method according to claim 1, wherein:
each of the earlier-processed data blocks include a first sub-data block and one or more second sub-data blocks, wherein: a second sub-data block of the one or more second sub-data blocks is an overlapping data block between an earlier-processed data block of the one or more earlier-processed data blocks and an adjacent data block of the earlier-processed data block; and a processing result of the second sub-data block is multiplexed by the adjacent data block.
Limitation(s) 7 a amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 7 and therefore inherits its rejections.
8. The method according to claim 7, further comprising:
processing the first sub-data block to obtain a first sub-processing result of the first sub-data block;
processing the one or more second sub-data blocks to obtain a second sub-processing result of each second sub-data block of the one or more second sub-data blocks;
and storing each second sub-processing result into a multiplexable data buffer area corresponding to the second sub-processing result.
Limitation(s) 8 a,b can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
Limitation(s) 8 c amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 8 and therefore inherits its rejections.
9. The method according to claim 8, wherein:
the later-processed data block includes a third sub-data block and one or more fourth sub-data blocks, wherein each of the one or more fourth sub-data blocks is a target sub-data block overlapping with the second sub-data block in the earlier-processed data block adjacent to the later-processed data block.
Limitation(s) 9 a amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 9 and therefore inherits its rejections.
10. The method according to claim 9, wherein processing the later-processed data block includes:
processing the third sub-data block to obtain a third sub-processing result of the third sub-data block;
acquiring, from a corresponding storage area in the multiplexable data buffer area, a second sub-processing result corresponding to a second sub-data block of the one or more second sub-data block overlapping with each of the target sub-data blocks;
and merging the third sub-processing result and the second sub-processing result of each target sub-data block to obtain the processing result of the later-processed data block.
Limitation(s) 10 a-c can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 10 and therefore inherits its rejections.
11. The method according to claim 10, wherein:
when the later-processed data block is processed
using a first network layer of the neural network
the method further includes acquiring the third sub-data block
from an off-chip memory.
Limitation(s) 11 a can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
Limitation(s) 11 b,d provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer.
Limitation(s) 11 c amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 1 and therefore inherits its rejections.
12. The method according to claim 1, wherein:
the neural network includes a plurality of network layers,
and the plurality of data blocks include a plurality of initial data blocks;
and sequentially processing the plurality of data blocks includes: for each initial data block of the plurality of initial data blocks, using the plurality of network layers to process the initial data block in sequence to obtain a processing result corresponding to the initial data block at each of the plurality of network layers,
wherein the processing results corresponding to adjacent initial data blocks of the plurality of initial data blocks at each of the plurality of network layers have overlapping data.
Limitation(s) 12 c can be performed in the human mind through observation, evaluation, judgement and opinion, with the aid of pen and paper, and is/are therefore reciting a mental process.
Limitation(s) 12 a provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer.
Limitation(s) 12 b,d c amount(s) to insignificant extra-solution activity of necessary data gathering/outputting.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
13. A data processing device, comprising:
a determination module,
configured to determine a target processing order for processing a plurality of data blocks in data, based on maximum storage capacity of a buffer,
wherein the data is divided into the plurality of data blocks for separate processing, and adjacent data blocks in the plurality of data blocks have overlapping data;
and a processing module,
configured to sequentially process the plurality of data blocks based (…) on the target processing order,
using a neural network
wherein at least a part of a processing result of a later-processed data block of the plurality of data blocks is a multiplexing of a processing result corresponding to the overlapping data between the later-processed data block and one or more earlier-processed data blocks of the plurality of data blocks, adjacent to the later-processed data block.
Limitation(s) 13 a,d provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer.
The remaining limitation(s) correspond(s) to those of claim 1 and therefore inherit(s) the same rejections.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 13 and therefore inherits its rejections.
14. An electronic device
comprising one or more processors configured to execute the data processing device according to claim 13.
Limitation(s) 14 provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 14 and therefore inherits its rejections.
15. The device according to claim 14,
wherein the neural network includes one or more network layers,
and the one or more processors are further configured to perform: determining a multiplexable data amount corresponding to a process result multiplexed by a data block and an adjacent data block of the data block, in the plurality of data blocks, wherein the multiplexable data amount represents a sum of data amounts corresponding to the processing result of the overlapping data corresponding to the data block and the adjacent data block at each of the one or more network layers;
dividing the buffer into one or more multiplexable data buffer areas, based on the maximum storage capacity and the multiplexable data amount;
and determining the target processing order for processing the plurality of data blocks in the data, based on a quantity of the one or more multiplexable data buffer areas.
The limitation(s) correspond(s) to those of claim 2 and therefore inherit(s) the same rejections.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 15 and therefore inherits its rejections.
16. The device according to claim 15, wherein:
the adjacent data blocks include: data blocks of the plurality of data blocks, adjacent along a first direction, and data blocks of the plurality of data blocks, adjacent along a second direction, wherein the first direction is perpendicular to the second direction; and the multiplexable data amount includes: a first multiplexable data amount between the data blocks adjacent along the first direction, and a second multiplexable data amount between the data blocks adjacent along the second direction.
The limitation(s) correspond(s) to those of claim 3 and therefore inherit(s) the same rejections.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 16 and therefore inherits its rejections.
17. The device according to claim 16,
wherein the one or more processors are further configured to perform: dividing an on-chip buffer of the buffer into one or more first-type multiplexable data buffer areas and one or more second-type multiplexable data buffer areas, based on the maximum storage capacity, the first multiplexable data amount and the second multiplexable data amount, wherein:
the one or more first-type multiplexable data buffers are configured to store a processing result corresponding to an overlapping data between two data blocks of the plurality of data blocks, adjacent along the first direction; and the one or more second-type multiplexable data buffer areas are configured to store a processing result corresponding to an overlapping data between two data blocks of the plurality of data blocks, adjacent along the second direction.
The limitation(s) correspond(s) to those of claim 4 and therefore inherit(s) the same rejections.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 15 and therefore inherits its rejections.
18. The device according to claim 15,
wherein the neural network includes a convolutional neural network,
and the one or more processors are further configured to perform: determining the multiplexable data amount, based on a quantity of convolutional layers of the convolutional neural network, a size of a convolutional kernel, and a preset output data size corresponding to the data,
wherein the preset output data size is characterized by a data size corresponding to a final processing result obtained through neural network processing based on a preset data size.
The limitation(s) correspond(s) to those of claim 5 and therefore inherit(s) the same rejections.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
The following claim inherits from claim 15 and therefore inherits its rejections.
19. The device according to claim 15,
wherein the one or more processors are further configured to perform: determining a data multiplexing rate corresponding to each processing order of processing the plurality of data blocks based on the quantity of the one or more multiplexable data buffer areas,
wherein the data multiplexing rate represents a multiplexing ratio of the processing result of the overlapping data in the plurality of data blocks;
and determining the target processing order based on the data multiplexing rate corresponding to each processing order.
The limitation(s) correspond(s) to those of claim 6 and therefore inherit(s) the same rejections.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
20. A non-transitory computer readable storage medium containing computer program that, when being executed,
causes at least one processor to perform:
determining a target processing order for processing a plurality of data blocks in data, based on maximum storage capacity of a buffer,
wherein the data is divided into the plurality of data blocks for separate processing, and adjacent data blocks in the plurality of data blocks have overlapping data;
and sequentially processing the plurality of data blocks using a (…) based on the target processing order,
using a neural network
wherein at least a part of a processing result of a later-processed data block of the plurality of data blocks is a multiplexing of a processing result corresponding to the overlapping data between the later-processed data block and one or more earlier-processed data blocks of the plurality of data blocks, adjacent to the later-processed data block.
Limitation(s) 20 a provide(s) mere instructions to implement the limitations which can be performed in the human mind, i.e., the judicial exception, on a computer.
The remaining limitation(s) correspond(s) to those of claim 1 and therefore inherit(s) the same rejections.
An integration into a practical application is not substantiated from the above. Thus, the claim is
not eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1,13,14,20 is/are rejected under 35 U.S.C. 103 as being unpatentable over CN 1968084 A, NAGASAWA, 2007-05-23 in view of CN 115481717 A, XU, 2022-12-16, CN 111488116 B , Zhongke, 2022-11-22, and CN 116737237 A, Jiao, 2023-09-12.
Claim(s) 1,13,14,20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230186045 A1, Symes, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12.
Symes teaches the following.
1. A data processing method, comprising:
determining a target processing order for processing a plurality of data blocks in data, based on maximum storage capacity of a buffer,
[0023] A first embodiment, the technology described herein comprises a method of operating a data processing system, the data processing system comprising a main memory operable to store data for neural network processing, and a processor operable to execute neural network processing, the processor operable to execute neural network processing having an associated local storage operable to store data for neural network processing when the processor is performing neural network processing, the method comprising:
[0024] performing neural network processing comprising a sequence of operations to process an initial input data array for the sequence of operations to generate a final output data array of the sequence of operations for respective blocks of the initial input data array on a block-by-block basis, such that the initial input data array is processed as one or more blocks which are each processed independently according to the sequence of operations to generate a corresponding block of the final output data array;
[0071] Thus, in an embodiment, all outputs from operations in the sequence of operations that are to be used as an input for a subsequent operation in the sequence of operations will be stored in the single, local storage, chaining buffer, to be (and be) provided therefrom as an input to another (a subsequent) operation in the sequence of operations.
[0083] Correspondingly, in an embodiment the local storage, chaining buffer includes capacity (and in an embodiment has set aside capacity) for storing (intermediate) data between (e.g. elementwise) operations in a sequence of operations when the processor is performing neural network processing in the manner of the technology described herein, and capacity (storage) set aside for (allocated for) storing data, such as, and in an embodiment, input feature map data, from main memory.
[0115] In an embodiment, the first operation in the sequence of operations is a more complex operation, with the subsequent operations in the sequence of operations being different and simpler operations. In an embodiment the subsequent operations after the first operation can all be performed and follow a particular, consistent, data processing order (such that they can be performed as a sequence of operations one after another and independently of each other), with the first operation then, e.g., and in an embodiment, defining the order in which the data will be processed by the subsequent operations. Thus in an embodiment, the first operation in the sequence of operations sets (defines) the order that the data will be processed over the sequence of operations (the data processing ordering), with the subsequent operations in the sequence of operations then following that data processing ordering established by the first operation in the sequence of operations (the subsequent operations in the sequence are, e.g. and in an embodiment, simpler, operations that can be executed and run in the data processing order (block/sub-block order) determined by the first operation).
wherein the data is divided into the plurality of data blocks for separate processing,
[0024] performing neural network processing comprising a sequence of operations to process an initial input data array for the sequence of operations to generate a final output data array of the sequence of operations for respective blocks of the initial input data array on a block-by-block basis, such that the initial input data array is processed as one or more blocks which are each processed independently according to the sequence of operations to generate a corresponding block of the final output data array;
(…) and sequentially processing the plurality of data blocks using a neural network based on the target processing order,
[0024] performing neural network processing comprising a sequence of operations to process an initial input data array for the sequence of operations to generate a final output data array of the sequence of operations for respective blocks of the initial input data array on a block-by-block basis, such that the initial input data array is processed as one or more blocks which are each processed independently according to the sequence of operations to generate a corresponding block of the final output data array;
Symes does not teach the following.
and adjacent data blocks in the plurality of data blocks have overlapping data;
Xu does however.
Page 10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Xu provides adjacent data blocks with overlapping data. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes as this combination would allow for identifying and minimizing redundant processing.
Symes does not teach the following.
wherein at least a part of a processing result of a later-processed data block of the plurality of data blocks is a multiplexing of a processing result corresponding to the (…) data between the later-processed data block and one or more earlier-processed data blocks of the plurality of data blocks, adjacent to the later-processed data block.
Jiao does however.
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
Jiao provides a multiplexed processing result as well as input (earlier processed) and output (later-processed) data blocks. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce these elements into the design of Symes in view of Xu as this combination would enable greater efficiency and throughput via the consolidation of data transmission.
The following claim corresponds to claim 1 and therefore inherits its rejection.
13. A data processing device, comprising: a determination module, configured to determine a target processing order for processing a plurality of data blocks in data, based on maximum storage capacity of a buffer, wherein the data is divided into the plurality of data blocks for separate processing, and adjacent data blocks in the plurality of data blocks have overlapping data; and a processing module, configured to sequentially process the plurality of data blocks using a neural network based on the target processing order, wherein at least a part of a processing result of a later-processed data block of the plurality of data blocks is a multiplexing of a processing result corresponding to the overlapping data between the later-processed data block and one or more earlier-processed data blocks of the plurality of data blocks, adjacent to the later-processed data block.
14. An electronic device
Symes teaches the following.
comprising one or more processors configured to execute the data processing device according to claim 13.
[0023] A first embodiment, the technology described herein comprises a method of operating a data processing system, the data processing system comprising a main memory operable to store data for neural network processing, and a processor operable to execute neural network processing, the processor operable to execute neural network processing having an associated local storage operable to store data for neural network processing when the processor is performing neural network processing, the method comprising:
The following claim corresponds to claim 1 and therefore inherits its rejection.
20. A non-transitory computer readable storage medium containing computer program that, when being executed, causes at least one processor to perform: determining a target processing order for processing a plurality of data blocks in data, based on maximum storage capacity of a buffer, wherein the data is divided into the plurality of data blocks for separate processing, and adjacent data blocks in the plurality of data blocks have overlapping data; and sequentially processing the plurality of data blocks using a neural network based on the target processing order, wherein at least a part of a processing result of a later-processed data block of the plurality of data blocks is a multiplexing of a processing result corresponding to the overlapping data between the later-processed data block and one or more earlier-processed data blocks of the plurality of data blocks, adjacent to the later-processed data block.
Claim(s) 2,15 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230186045 A1, Symes, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 6157674 A. Oda, 2000-12-05, and IT 1174015 B, Parziale, 1987-06-24.
2. The method according to claim 1,
Symes teaches the following.
wherein the neural network includes one or more network layers,
[0045] The technology described herein relates to neural network processing, and in particular to the performing of a sequence of operations (a sequence of layers of neural network processing) when performing neural network processing. In the technology described herein, a sequence of operations for neural network processing is performed on a block-by-block basis, i.e. such that the initial input data array to the sequence of operations is processed (is subjected to the sequence of operations) on a block-by-block basis.
and determining the target processing order for processing the plurality of data blocks in the data, based on the maximum storage capacity of the buffer, includes:
[0023] A first embodiment, the technology described herein comprises a method of operating a data processing system, the data processing system comprising a main memory operable to store data for neural network processing, and a processor operable to execute neural network processing, the processor operable to execute neural network processing having an associated local storage operable to store data for neural network processing when the processor is performing neural network processing, the method comprising:
[0024] performing neural network processing comprising a sequence of operations to process an initial input data array for the sequence of operations to generate a final output data array of the sequence of operations for respective blocks of the initial input data array on a block-by-block basis, such that the initial input data array is processed as one or more blocks which are each processed independently according to the sequence of operations to generate a corresponding block of the final output data array;
[0071] Thus, in an embodiment, all outputs from operations in the sequence of operations that are to be used as an input for a subsequent operation in the sequence of operations will be stored in the single, local storage, chaining buffer, to be (and be) provided therefrom as an input to another (a subsequent) operation in the sequence of operations.
[0083] Correspondingly, in an embodiment the local storage, chaining buffer includes capacity (and in an embodiment has set aside capacity) for storing (intermediate) data between (e.g. elementwise) operations in a sequence of operations when the processor is performing neural network processing in the manner of the technology described herein, and capacity (storage) set aside for (allocated for) storing data, such as, and in an embodiment, input feature map data, from main memory.
[0115] In an embodiment, the first operation in the sequence of operations is a more complex operation, with the subsequent operations in the sequence of operations being different and simpler operations. In an embodiment the subsequent operations after the first operation can all be performed and follow a particular, consistent, data processing order (such that they can be performed as a sequence of operations one after another and independently of each other), with the first operation then, e.g., and in an embodiment, defining the order in which the data will be processed by the subsequent operations. Thus in an embodiment, the first operation in the sequence of operations sets (defines) the order that the data will be processed over the sequence of operations (the data processing ordering), with the subsequent operations in the sequence of operations then following that data processing ordering established by the first operation in the sequence of operations (the subsequent operations in the sequence are, e.g. and in an embodiment, simpler, operations that can be executed and run in the data processing order (block/sub-block order) determined by the first operation).
(…) at each of the one or more network layers
[0045] The technology described herein relates to neural network processing, and in particular to the performing of a sequence of operations (a sequence of layers of neural network processing) when performing neural network processing. In the technology described herein, a sequence of operations for neural network processing is performed on a block-by-block basis, i.e. such that the initial input data array to the sequence of operations is processed (is subjected to the sequence of operations) on a block-by-block basis.
(…) based on the maximum storage capacity
[0071] Thus, in an embodiment, all outputs from operations in the sequence of operations that are to be used as an input for a subsequent operation in the sequence of operations will be stored in the single, local storage, chaining buffer, to be (and be) provided therefrom as an input to another (a subsequent) operation in the sequence of operations.
[0083] Correspondingly, in an embodiment the local storage, chaining buffer includes capacity (and in an embodiment has set aside capacity) for storing (intermediate) data between (e.g. elementwise) operations in a sequence of operations when the processor is performing neural network processing in the manner of the technology described herein, and capacity (storage) set aside for (allocated for) storing data, such as, and in an embodiment, input feature map data, from main memory.
Xu teaches the following.
(…) overlapping (…)
Page 10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Jiao teaches the following.
corresponding to a process result multiplexed by a data block and an adjacent data block of the data block, in the plurality of data blocks,
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
corresponding to the processing result of the (… ) data corresponding to the data block and the adjacent data block (…);
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
Symes in view of Xu and Jiao does not teach the following.
determining a multiplexable data amount
Oda does however.
At step 104 (S104), the scheduling circuit 128 calculates the amount (multiplexable amount)
(…) wherein the multiplexable data amount represents a sum of data amounts
in the third embodiment, the boundary of the video frame cycle and the boundary of the transport packet contained in the transport stream are brought into coincidence and the sum of the amounts (multiplexable amounts) of the compressed video data and the compressed audio data which are multiplexed by the audio and/or video data transmitting apparatus 1 for every video frame cycle and transmitted via the ATM communication line to the audio and/or video data receiving apparatus 2 is specified to a multiple of the amount of data that can be accommodated in the payload part of the transport packet.
(…) based on (…) the multiplexable data amount;
in the third embodiment, the boundary of the video frame cycle and the boundary of the transport packet contained in the transport stream are brought into coincidence and the sum of the amounts (multiplexable amounts) of the compressed video data and the compressed audio data which are multiplexed by the audio and/or video data transmitting apparatus 1 for every video frame cycle and transmitted via the ATM communication line to the audio and/or video data receiving apparatus 2 is specified to a multiple of the amount of data that can be accommodated in the payload part of the transport packet.
Oda provides the determination of a multiplexable data amount. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Jiao and Xu as this combination would enable greater efficiency and throughput via the consolidation of data transmission.
Symes in view of Xu and Jiao does not teach the following.
(…) dividing the buffer into one or more multiplexable data buffer areas,
Parziale does however.
Page 14, Paragraph 16: The memory matrix 95 is addressed through the address bus 91 with the aid of the area decoding bus26.Ing. Roberto DE NOVA ila proprio and par i olirli 92, which has the task of increasing the addressing capacity available on the address bus 91 by dividing the memory matrix 95 into homogeneous and multiplexable areas or banks.
(…) based on a quantity of the one or more multiplexable data buffer areas.
Page 14, Paragraph 16: The memory matrix 95 is addressed through the address bus 91 with the aid of the area decoding bus26.Ing. Roberto DE NOVA ila proprio and par i olirli 92, which has the task of increasing the addressing capacity available on the address bus 91 by dividing the memory matrix 95 into homogeneous and multiplexable areas or banks.
It is implicit that the quantity of the multiplexable areas would be obtained from the division.
Parizale provides dividing memory into multiplexable areas. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Jiao and Xu as this combination would enable greater efficiency and throughput via the consolidation of data transmission.
The following claim corresponds to claim 2 and therefore inherits its rejection.
15. The device according to claim 14, wherein the neural network includes one or more network layers, and the one or more processors are further configured to perform: determining a multiplexable data amount corresponding to a process result multiplexed by a data block and an adjacent data block of the data block, in the plurality of data blocks, wherein the multiplexable data amount represents a sum of data amounts corresponding to the processing result of the overlapping data corresponding to the data block and the adjacent data block at each of the one or more network layers; dividing the buffer into one or more multiplexable data buffer areas, based on the maximum storage capacity and the multiplexable data amount; and determining the target processing order for processing the plurality of data blocks in the data, based on a quantity of the one or more multiplexable data buffer areas.
Claim(s) 3,16 is/are rejected under 35 U.S.C. 103 as being unpatentable over CN 1968084 A, NAGASAWA, 2007-05-23 in view of CN 115481717 A, XU, 2022-12-16, CN 111488116 B , Zhongke, 2022-11-22, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 6157674 A. Oda, 2000-12-05, IT 1174015 B, Parziale, 1987-06-24, 20110187903 A1, Kim, 2010-09-13 and JP 3599593 B2, Kusakabe, 2004-12-08.
3. The method according to claim 2, wherein: the adjacent data blocks include:
Symes in view of Xu, Jiao, Oda and Paziale does not teach the following.
data blocks of the plurality of data blocks, adjacent along a first direction,
Kim does however.
[0069] A plurality of image data blocks, namely, first through third image data blocks B1, B2, B3, . . . are extracted from an input image signal. Referring to FIG. 6, the first image data block B1 and the second image data block B2 are disposed adjacent to each other in a row direction (an x-axis direction). In addition, the second image data block B2 and the third image data block B3 are disposed adjacent to each other in the row direction (the x-axis direction). However, the invention is not limited to the arrangement of the image data blocks shown in FIG. 6 and thus, for example, the first image data block B1 and the second image data block B2 may be adjacently disposed in the column direction (the y-axis direction).
and data blocks of the plurality of data blocks, adjacent along a second direction,
[0069] A plurality of image data blocks, namely, first through third image data blocks B1, B2, B3, . . . are extracted from an input image signal. Referring to FIG. 6, the first image data block B1 and the second image data block B2 are disposed adjacent to each other in a row direction (an x-axis direction). In addition, the second image data block B2 and the third image data block B3 are disposed adjacent to each other in the row direction (the x-axis direction). However, the invention is not limited to the arrangement of the image data blocks shown in FIG. 6 and thus, for example, the first image data block B1 and the second image data block B2 may be adjacently disposed in the column direction (the y-axis direction).
wherein the first direction is perpendicular to the second direction;
[0069] A plurality of image data blocks, namely, first through third image data blocks B1, B2, B3, . . . are extracted from an input image signal. Referring to FIG. 6, the first image data block B1 and the second image data block B2 are disposed adjacent to each other in a row direction (an x-axis direction). In addition, the second image data block B2 and the third image data block B3 are disposed adjacent to each other in the row direction (the x-axis direction). However, the invention is not limited to the arrangement of the image data blocks shown in FIG. 6 and thus, for example, the first image data block B1 and the second image data block B2 may be adjacently disposed in the column direction (the y-axis direction).
Kim provides adjacent blocks laid out along perpendicular axes. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce these elements into the design of Nagasawa in view of Xu, Zhongke, Jiao, Oda and Paziale as this combination would be adapted to better handle processing that requires a two-dimensional organization of data, such as image processing.
Symes in view of Xu, Jiao, Oda and Paziale does not teach the following.
and the multiplexable data amount includes: a first multiplexable data amount between the data blocks adjacent along the first direction,
Kusakabe does however.
[0032] In step S300, variables i and j and the count value of the multiplexable information amount are initialized. Here, a variable i is a variable for counting an address (pixel position) in the vertical (row) direction, and a variable j is a variable for counting an address (pixel position) in the horizontal (column) direction, and is indicated by variables i and j. The position of the pixel is called a row address and a column address, respectively.
and a second multiplexable data amount between the data blocks adjacent along the second direction.
[0032] In step S300, variables i and j and the count value of the multiplexable information amount are initialized. Here, a variable i is a variable for counting an address (pixel position) in the vertical (row) direction, and a variable j is a variable for counting an address (pixel position) in the horizontal (column) direction, and is indicated by variables i and j. The position of the pixel is called a row address and a column address, respectively.
Kusakabe provides counting (determining an amount of) the multiplexable information (data) laid out along perpendicular axes. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce these elements into the design of Nagasawa in view of Xu, Zhongke, Jiao, Oda and Paziale as this combination would enable greater efficiency and throughput via the consolidation of data transmission.
The following claim corresponds to claim 3 and therefore inherits its rejection.
16. The device according to claim 15, wherein: the adjacent data blocks include: data blocks of the plurality of data blocks, adjacent along a first direction, and data blocks of the plurality of data blocks, adjacent along a second direction, wherein the first direction is perpendicular to the second direction; and the multiplexable data amount includes: a first multiplexable data amount between the data blocks adjacent along the first direction, and a second multiplexable data amount between the data blocks adjacent along the second direction.
Claim(s) 4,17 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230186045 A1, Symes, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 6157674 A. Oda, 2000-12-05, IT 1174015 B, Parziale, 1987-06-24, 20110187903 A1, Kim, 2010-09-13, JP 3599593 B2, Kusakabe, 2004-12-08, and US 20130194286 A1, Bourd, 2013-01-23.
4. The method according to claim 3, wherein dividing the buffer into the one or more multiplexable data buffer areas, based on the maximum storage capacity and the multiplexable data amount, includes:
Symes teaches the following.
(…) based on the maximum storage capacity,
[0071] Thus, in an embodiment, all outputs from operations in the sequence of operations that are to be used as an input for a subsequent operation in the sequence of operations will be stored in the single, local storage, chaining buffer, to be (and be) provided therefrom as an input to another (a subsequent) operation in the sequence of operations.
[0083] Correspondingly, in an embodiment the local storage, chaining buffer includes capacity (and in an embodiment has set aside capacity) for storing (intermediate) data between (e.g. elementwise) operations in a sequence of operations when the processor is performing neural network processing in the manner of the technology described herein, and capacity (storage) set aside for (allocated for) storing data, such as, and in an embodiment, input feature map data, from main memory.
Xu teaches the following.
(…) corresponding to an overlapping data between two data blocks of the plurality of data blocks Page
10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Parziale teaches the following.
dividing (…) into one or more first-type multiplexable data buffer areas and one or more second-type multiplexable data buffer areas,
Page 14, Paragraph 16: The memory matrix 95 is addressed through the address bus 91 with the aid of the area decoding bus26.Ing. Roberto DE NOVA ila proprio and par i olirli 92, which has the task of increasing the addressing capacity available on the address bus 91 by dividing the memory matrix 95 into homogeneous and multiplexable areas or banks.
Kim teaches the following.
wherein: the one or more first-type multiplexable data buffers are configured to store a processing result (…), adjacent along the first direction;
and the one or more second-type multiplexable data buffer areasare configured to store a processing result (…), adjacent along the second direction.
[0069] A plurality of image data blocks, namely, first through third image data blocks B1, B2, B3, . . . are extracted from an input image signal. Referring to FIG. 6, the first image data block B1 and the second image data block B2 are disposed adjacent to each other in a row direction (an x-axis direction). In addition, the second image data block B2 and the third image data block B3 are disposed adjacent to each other in the row direction (the x-axis direction). However, the invention is not limited to the arrangement of the image data blocks shown in FIG. 6 and thus, for example, the first image data block B1 and the second image data block B2 may be adjacently disposed in the column direction (the y-axis direction).
Kusakabe teaches the following.
based on (…) the first multiplexable data amount and the second multiplexable data amount,
[0032] In step S300, variables i and j and the count value of the multiplexable information amount are initialized. Here, a variable i is a variable for counting an address (pixel position) in the vertical (row) direction, and a variable j is a variable for counting an address (pixel position) in the horizontal (column) direction, and is indicated by variables i and j. The position of the pixel is called a row address and a column address, respectively.
Symes in view of Xu, Jiao, Oda Paziale, Kim, and Kusakabe does not teach the following.
an on-chip buffer of the buffer
Bourd does however.
[0095] Buffers 42 and optional on-chip buffers 38 may assist with organizing the workload scheduling for programmable compute units 28 and fixed-function compute units 32.
It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Nagasawa in view of Xu, Zhongke, Jiao, Oda Paziale, Kim, and Kusakabe as a buffer physically located on the same die as the processing units would confer minimal propagation delays.
The following claim corresponds to claim 4 and therefore inherits its rejection.
17. The device according to claim 16, wherein the one or more processors are further configured to perform: dividing an on-chip buffer of the buffer into one or more first-type multiplexable data buffer areas and one or more second-type multiplexable data buffer areas, based on the maximum storage capacity, the first multiplexable data amount and the second multiplexable data amount, wherein: the one or more first-type multiplexable data buffers are configured to store a processing result corresponding to an overlapping data between two data blocks of the plurality of data blocks, adjacent along the first direction; and the one or more second-type multiplexable data buffer areas are configured to store a processing result corresponding to an overlapping data between two data blocks of the plurality of data blocks, adjacent along the second direction.
Claim(s) 5,18 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230186045 A1, Symes, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 6157674 A. Oda, 2000-12-05, IT 1174015 B, Parziale, 1987-06-24, US 20230072708 A1, Wang, 2022-03-23, and US 20210045675 A1, Lee, 2020-01-16.
5. The method according to claim 2,
Symes teaches the following.
wherein the neural network includes a convolutional neural network,
[0092] The overall neural network that the operation in the manner of the technology described herein is used with will, as will be appreciated by those skilled in the art, comprise a sequence of neural network processing operations (layers). The overall neural network may comprise any suitable and desired number and types of layers, such as convolution layers, activation layers, pooling layers, etc.
Jiao teaches the following.
corresponding to the process result multiplexed by the data block and the adjacent data block of the data block, in the plurality of data blocks, includes:
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
Oda teaches the following.
(…) and determining the multiplexable data amount (…)
At step 104 (S104), the scheduling circuit 128 calculates the amount (multiplexable amount)
determining the multiplexable data amount,
At step 104 (S104), the scheduling circuit 128 calculates the amount (multiplexable amount)
Symes in view of Xu, Jiao, Oda, and Paziale does not teach the following.
based on a quantity of convolutional layers of the convolutional neural network, a size of a convolutional kernel, and a (…) output data size corresponding to the data, wherein the (…) output data size is characterized by a data size corresponding to a final processing result obtained through neural network processing based on a (…) data size.
Wang does however.
[0027] Further, the multi-scale feature fusion module in the Step S2 includes a 1-dimension CNN (1D-CNN) with three convolutional layers, in which the kernel size is k × 1, and the corresponding output is a 1 × 1 × d × k tensor. The output of each convolutional layer is flattened into a 1 × 1 × q tensor, where q = d × k;
Wang provides a calculation based on the quantity of convolutional layers, kernel size, and output size where the output is a result of processing by the layers of NN. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Xu,, Jiao, Oda and Paziale as this combination would allow consistent metrics to be obtained for variation in the different elements of the system, such as the number of layers or the size of the kernel.
Symes in view of Xu, Jiao, Oda, and Paziale does not teach the following.
Preset data size
Lee does however.
[0045] The defining of the statistical sequence data may include transforming the signal data processed in a form of the time series data to a preset data size and forwarding the transformed signal data to the 1D convolution layer;
Lee provides preset data in the context of neural network processing. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Xu,, Jiao, Oda and Paziale as this combination would allow consistent metrics to be obtained for variation in the different elements of the system, such as the number of layers or the size of the kernel.
The following claim corresponds to claim 5 and therefore inherits its rejection.
18. The device according to claim 15, wherein the neural network includes a convolutional neural network, and the one or more processors are further configured to perform: determining the multiplexable data amount, based on a quantity of convolutional layers of the convolutional neural network, a size of a convolutional kernel, and a preset output data size corresponding to the data, wherein the preset output data size is characterized by a data size corresponding to a final processing result obtained through neural network processing based on a preset data size.
Claim(s) 6,19 is/are rejected under 35 U.S.C. 103 as being unpatentable over over US 20230186045 A1, Symes, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 6157674 A. Oda, 2000-12-05, IT 1174015 B, Parziale, 1987-06-24, US 20230072708 A1, US 20210373968 A1, Iwasa, 2019-10-29 and FR 2731127 A1, Kimura, 1996-08-30.
6. The method according to claim 2, wherein determining the target processing order for processing the plurality of data blocks in the data, based on the quantity of the one or more multiplexable data buffer areas, includes:
Symes teaches the following.
(…) corresponding to each processing order of processing the plurality of data blocks
[0115] In an embodiment, the first operation in the sequence of operations is a more complex operation, with the subsequent operations in the sequence of operations being different and simpler operations. In an embodiment the subsequent operations after the first operation can all be performed and follow a particular, consistent, data processing order (such that they can be performed as a sequence of operations one after another and independently of each other), with the first operation then, e.g., and in an embodiment, defining the order in which the data will be processed by the subsequent operations. Thus in an embodiment, the first operation in the sequence of operations sets (defines) the order that the data will be processed over the sequence of operations (the data processing ordering), with the subsequent operations in the sequence of operations then following that data processing ordering established by the first operation in the sequence of operations (the subsequent operations in the sequence are, e.g. and in an embodiment, simpler, operations that can be executed and run in the data processing order (block/sub-block order) determined by the first operation).
Xu teaches the following.
of the overlapping data in the plurality of data blocks
Page 10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Parziale teaches the following.
based on the quantity of the one or more multiplexable data buffer areas,
Page 14, Paragraph 16: The memory matrix 95 is addressed through the address bus 91 with the aid of the area decoding bus26.Ing. Roberto DE NOVA ila proprio and par i olirli 92, which has the task of increasing the addressing capacity available on the address bus 91 by dividing the memory matrix 95 into homogeneous and multiplexable areas or banks.
Symes in view of Xu, Jiao, Oda, and Paziale does not teach the following.
determining a data multiplexing rate (…) wherein the data multiplexing rate represents a multiplexing ratio of the processing result(…);
Iwasa does however.
[0062] The “multiplexing rate” is the ratio (CPU performance ratio) of the CPU performance of each compute 30 to the CPU performance of a CPU serving as a reference,
Iwasa provides a determination of a data multiplexing rate representing a ratio of the processing result. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Nagasawa in view of Xu, Zhongke, Jiao, Oda and Paziale as this combination would allow data to be transmitted according to the processing output, scaling resource use to its actual need.
Nagasawa in view of Xu, Zhongke, Jiao, Oda, and Paziale does not teach the following.
and determining the target processing order based on the data multiplexing rate corresponding to each processing order.
Kimura does however.
A synchronization controller 13 is provided to receive the outputs of the transmission control processors 10, in order to determine the quantity of control data to be transmitted, and controls the buffer memories 11 to read the control signals stored in the multiplexer 12 in depending on the detected amount of data.
Here, a synchronization controller provided to receive the outputs of the transmission control processors 10, in order to determine the quantity of control data to be transmitted corresponds to a processing order. Furthermore, controlling the buffer memories to read the control signals stored in the multiplexer in depending on the detected amount of data corresponds to the multiplexing rate corresponding to each processing order.
Kimura provides determining a processing order based on a the multiplexing rate corresponding to each processing order. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce these elements into the design of Nagasawa in view of Xu, Zhongke, Jiao, Oda and Paziale as this combination would allow a processing sequence to be established which emphasizes reuse, bolstering system efficiency.
The following claim corresponds to claim 6 and therefore inherits its rejection.
19. The device according to claim 15, wherein the one or more processors are further configured to perform: determining a data multiplexing rate corresponding to each processing order of processing the plurality of data blocks based on the quantity of the one or more multiplexable data buffer areas, wherein the data multiplexing rate represents a multiplexing ratio of the processing result of the overlapping data in the plurality of data blocks; and determining the target processing order based on the data multiplexing rate corresponding to each processing order.
Claim(s) 7,8,9 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230186045 A1, SYMES, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 20120023384 A1, Naradasi, 2010-09-15.
7. The method according to claim 1, wherein:
Xu teaches the following.
wherein: a (…) block of the one or more (…) blocks is an overlapping data block between an earlier-processed data block of the one or more earlier-processed data blocks and an adjacent data block of the earlier-processed data block;
Page 10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Jiao teaches the following.
each of the earlier-processed data blocks (…)
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
and a processing result of the (…) block is multiplexed by the adjacent data block.
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
Symes in view of Xu, and Jiao does not teach the following.
(…) include a first sub-data block and one or more second sub-data blocks,
Naradasi does however.
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Xu, Zhongke, and Jiao as this combination would allow for more granular partitioning of data, meaning the system can process only relevant sub blocks without needing to waste resources filtering through larger data sets.
8. The method according to claim 7, further comprising:
Symes teaches the following.
and storing each (…) processing result into a (…) data buffer area corresponding to the (…) result.
[0071] Thus, in an embodiment, all outputs from operations in the sequence of operations that are to be used as an input for a subsequent operation in the sequence of operations will be stored in the single, local storage, chaining buffer, to be (and be) provided therefrom as an input to another (a subsequent) operation in the sequence of operations.
Jiao teaches the following.
processing the (…) data block to obtain a (…) -processing result of the (…) block; processing the one or more (…) data blocks to obtain (…) processing result of each (…) data block of the one or more (…) data blocks;
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
Parziale teaches the following.
multiplexable data buffer
Page 14, Paragraph 16: The memory matrix 95 is addressed through the address bus 91 with the aid of the area decoding bus26.Ing. Roberto DE NOVA ila proprio and par i olirli 92, which has the task of increasing the addressing capacity available on the address bus 91 by dividing the memory matrix 95 into homogeneous and multiplexable areas or banks.
Naradasi teaches the following.
first sub-data block
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
second sub-data block
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
9. The method according to claim 8,
Xu teaches the following.
is a target sub-data block (…) overlapping with the (…) block in the earlier-processed data block adjacent to the later-processed data block.
Page 10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Naradasi teaches the following.
wherein each of the one or more fourth sub-data blocks
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
second sub-data
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
Symes in view of Xu, and Jiao does not teach the following.
wherein: the later-processed data block includes a third sub-data block and one or more fourth sub-data blocks,
Naradasi does however.
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Xu, and Jiao as this combination would allow for more granular partitioning of data, meaning the system can process only relevant sub blocks without needing to waste resources filtering through larger data sets.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230186045 A1, Symes, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 20120023384 A1, Naradasi, 2010-09-15, JP H11232294 A , Yamazaki, 1999-08-27 and CN 116777376 A, Huang, 2023-09-19.
10. The method according to claim 9, wherein processing the later-processed data block includes:
Symes the following.
processing the (…) block to obtain a (…) processing result of the (…) block;
[0024] performing neural network processing comprising a sequence of operations to process an initial input data array for the sequence of operations to generate a final output data array of the sequence of operations for respective blocks of the initial input data array on a block-by-block basis, such that the initial input data array is processed as one or more blocks which are each processed independently according to the sequence of operations to generate a corresponding block of the final output data array;
of the later-processed data block.
[0024] performing neural network processing comprising a sequence of operations to process an initial input data array for the sequence of operations to generate a final output data array of the sequence of operations for respective blocks of the initial input data array on a block-by-block basis, such that the initial input data array is processed as one or more blocks which are each processed independently according to the sequence of operations to generate a corresponding block of the final output data array;
Xu teaches the following.
corresponding to (…) block of the one or more second (…) block overlapping with each of the target (…) blocks;
Page 10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Jiao teaches the following.
multiplexable
Page 4, Paragraph 3: Compared with the prior art, the invention has the following beneficial effects: The invention provides a neural network accelerator data reuse framework based on instruction control, through instruction analysis, decoding and multi-level transmission, controlling to realize the processing of the input block data according to the order of channel priority, capable of performing on-chip multiplexing on the part and data output by the adjacent data block, using the mode of instruction transfer, providing operator configuration information, the input cache area can generate the access address of the data block on the current chip according to this, and according to the instruction control information, the operator part and the data coverage cache, the accumulation cache, judging the result output, caching the result information of the data block on the chip, realizing the multiplexing of the adjacent data block part and the data in the channel accumulation, and finishing the output after calculating the output characteristic image block data.
Naradasi teaches the following.
third sub-data block
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
second sub (…)
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
of each target sub-data block
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
Symes in view of Xu, Jiao and Nardasi does not teach the following.
acquiring, from a corresponding storage area in the (…) data buffer area, a (…) processing result
Yamazaki does however.
Page 7, Paragraph 11: A processing unit for stacking request contents in the transaction queue and notifying reception, and in the case of referring to a processing result, acquiring a corresponding processing result from the processing result buffer and notifying the processing result;
Yamazaki provides the acquisition of a processing result from a storage buffer. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Xu, Jiao and Nardasi as data retrieval from a storage device amounts to a standard practice within the art.
Symes in view of Xu, Jiao and Nardasi does not teach the following.
and merging the (…) processing result and the (…) processing result (…) to obtain the processing result (…)
Huang does however.
Page 4, Paragraph 10: a combination sub-unit for combining each sub-processing result to obtain the final processing result
Huang provides the combination of sub processing results to obtain a final processing result. It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Xu, Jiao and Nardasi as this combination would allow for more granular partitioning of data, meaning the system can process only relevant sub blocks without needing to waste resources filtering through larger data sets.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230186045 A1, Symes, 2022-10-05 in view of CN 115481717 A, XU, 2022-12-16, and CN 116737237 A, Jiao, 2023-09-12, as well as in further view of US 20120023384 A1, Naradasi, 2010-09-15, JP H11232294 A , Yamazaki, 1999-08-27 CN 116777376 A, Huang, 2023-09-19 and US 20130194286 A1, Bourd, 2013-01-23.
11. The method according to claim 10, wherein: when the later-processed data block is processed using a first network layer of the neural network,
Nardasi teaches the following
the third sub-data block
[0015] A representative ECC page 120 is stored in the memory 112. The ECC page 120 includes a data block 130 that includes a first sub-block (SB1) 121, a second sub-block (SB2) 122, a third sub-block (SB3) 123, and a fourth sub-block (SB4) 124.
Symes in view of Xu, Jiao Nardasi, Yamazaki, and Huang does not teach the following.
the method further includes acquiring (…) from an off-chip memory.
Bourd does however.
[0054] Accordingly, in some examples, to execute kernels in a pipeline fashion, GPU 16 may retrieve produced data stored in a local memory of a programmable compute unit and store the produced data in global memory 20. Storing data in global memory 20 may be referred to a storing data off-chip because global memory 20 is external to the integrated circuit that houses GPU 16 (i.e., external to IC 12). GPU 16 may then retrieve the data stored in global memory 20 and load the retrieved data into the local memory of another programmable compute unit.
It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Nagasawa in view of Xu, Zhongke, Jiao, Nardasi, Yamazaki, and Huang as this combination would enable a continuity of functionality of the system when on chip memory is at capacity.
12. The method according to claim 1, wherein:
Symes teaches the following.
the neural network includes a plurality of network layers,
[0045] The technology described herein relates to neural network processing, and in particular to the performing of a sequence of operations (a sequence of layers of neural network processing) when performing neural network processing. In the technology described herein, a sequence of operations for neural network processing is performed on a block-by-block basis, i.e. such that the initial input data array to the sequence of operations is processed (is subjected to the sequence of operations) on a block-by-block basis.
and the plurality of data blocks include a plurality of (…) blocks;
[0024] performing neural network processing comprising a sequence of operations to process an initial input data array for the sequence of operations to generate a final output data array of the sequence of operations for respective blocks of the initial input data array on a block-by-block basis, such that the initial input data array is processed as one or more blocks which are each processed independently according to the sequence of operations to generate a corresponding block of the final output data array;
Xu teaches the following.
wherein the processing results corresponding to adjacent (…) blocks of the plurality of (…) data blocks at each of the plurality of network layers have overlapping data.
Page 10, Paragraph 2: The method specifically comprises: the electronic device 100 determines the size of each data block according to the size of each sub-result, then determining the previous data block of each data block in different dimensional direction, according to the size of the previous data block in different dimensional direction, the size of the sub-result corresponding to the previous data block and the model parameter of each layer of convolution layer, calculating the overlapping area of the data block and the previous data block in the input of the last layer of convolution layer in different dimensional direction, The size of the overlapping area of the adjacent data blocks in the input (i.e., input data) of the first layer of convolutional layer is then determined from the last layer of convolutional layer sequentially through different dimensional direction, overlapping area between the data block and the previous data block, then according to the overlapping area between the data block and the previous data block in different dimensional direction, determining the initial position of the data block corresponding to the sub-result in the input data, so as to determine the range of each data block.
Symes in view of Xu, Jiao does not teach the following.
and sequentially processing the plurality of data blocks includes: for each initial data block of the plurality of initial data blocks, using the plurality of network layers to process the initial data block in sequence to obtain a processing result corresponding to the initial data block at each of the plurality of network layers,
Chen, Ke-Lin does however.
Page 3, Paragraph 3: In some embodiments of the first aspect of the present application, the process of processing the current layer input data block located at the first position comprises: reading the initial block address and the stop block address of the rectangular block corresponding to the current layer input data block located at the first bit from the new block table; reading the cache idle table, according to sequence, distributing a block of idle cache for each previous layer of output data block, the initial block address of the cache block is set to be non-idle, updating the cache idle table, and writing the mapping relationship between each previous layer of output data block and its starting block address in the new-blk_addr_map register; sending the reading request to the external memory, writing the data of the previous layer of output data block into the buffer address of the block.
It would have been obvious to one skilled in the art at the time of this application’s filing to introduce this element into the design of Symes in view of Xu, and Jiao as establishing a processing sequence via a deeper neural network would enable the reuse of computations from earlier layers, improving system efficiency.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Luke Absher whose telephone number is (571) 270-1057. The examiner can normally be reached M-F: 8:00 am - 4:00 pm. Examiner interviews are available via telephone, in person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the US PTO Automated Interview Request (AIR) at http:/ /www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kevin Young can be reached at 571-270-3180.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https:/ /patentcenter.uspto.gov. Visit https:/ /www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786 9199 (IN USA OR CANADA) or 571-272-1000.
/LUCAS DONALD ABSHER/
Examiner, Art Unit 2194
/KEVIN L YOUNG/Supervisory Patent Examiner, Art Unit 2194
Prosecution Insights