DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/03/2025 has been entered.
Double Patenting
Double patenting analysis has been performed with respect to US App. No. 17/502,218 having common relationship of inventorship and/or ownership with respect to the pending claims, none are found to warrant a double patenting rejection. The claims of US App. No. 17/502,218 have been considered, yet they appear of different scope than the claims of this application.
Response to Arguments
Applicant's arguments filed 09/03/2025 have been fully considered and they are partially persuasive.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC § 103, the arguments are directed to newly amended limitations that were not previously examined by the examiner. Therefore, applicants arguments are rendered moot. The examiner refers to the rejection under 35 USC § 103 in the current office action for more details.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-7, 13-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US Pub No. US20180285727A1 Baum et al. (“Baum”) in view of US Pub No. US20190220742A1 Kuo et al. (“Kuo”)
In regards to claim 1,
Baum teaches An artificial intelligence accelerator comprising a control unit ie layer controller 82, a computing engine ie processing core 60, a group control unit ie NN Manager 62, and a group cache unit ie subcluster 70
(Baum, “[0101] A high-level block diagram illustrating an example NN processing core in more detail is shown in FIG. 5. The NN processing engine or core 60 [a computing engine] comprises several hierarchical computation units. The lowest hierarchical level is the processing element (PE) 76 with its own dedicated internal Layer 1 or L1 memory 78 in which individual neurons are implemented. A plurality of N PEs 76 along with dedicated Layer 2 or L2 memory 74 make up the next hierarchical level termed a subcluster 70 [a group cache unit]. A plurality of M subclusters 70 along with dedicated Layer 3 or L3 memory 72, a plurality of activation function circuits 80, and a plurality of layer controller (LC) [a control unit] circuits 82 make up a cluster 66. A plurality of L clusters along with dedicated Layer 4 or L4 memory 64 are in the NN processor core 60 which also comprises NN manager circuit 62 [group control unit], and memory interface 68 to off-chip Layer 5 or L5 memory 98. A plurality of bus interfaces 86 (i.e. chip-to-chip interfaces) couple the NN processor to other off-chip NN processor chips for additional network capacity. Bus interface 84 (i.e. chip-to-chip interface) couples the NN processor to a conventional rule based machine (RBM) co-processor 88 comprising a CPU 90, instruction memory 92 and data memory 94. In an alternative embodiment, the RBM co-processor is optionally coupled to the NN device 60 via a suitable interface, e.g., GPUs, I2C, etc.”)
PNG
media_image1.png
466
663
media_image1.png
Greyscale
Baum teaches and the artificial intelligence accelerator being executed on a hardware platform including at least one of a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC);
(Baum, “[0074] The invention is operational with numerous general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, cloud computing, hand-held or laptop devices, multiprocessor systems, microprocessor, microcontroller or microcomputer based systems, set top boxes, programmable consumer electronics, ASIC or FPGA core, DSP core, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.”)
Baum teaches the control unit being configured to parse a processing instruction for a target network layer in a neural network model to obtain a concurrent instruction,
Examiner interprets concurrent instruction in light of the specification, ([0039], “The concurrent instruction instructs to perform parallel processing on the input tile.”)
(Baum, “[0209] A high-level block diagram illustrating an example layer controller in more detail is shown in FIG. 19. The layer controller (LC), generally referenced 310, comprises a layer control unit (LCU) 314 responsible for decoding and executing microcode instructions 311 read from instruction memory 312 [the control unit being configured to parse a processing instruction for a target network layer in a neural network model…]. Depending on the instruction one or more command signals 313 are output to various control and decode blocks, including input aligner control 316, activation control 318, input address decoder 320, weight address decoder 322, output address decoder 324, and PE control 326. The control and address signals from these six blocks are respectively output to input aligner 328, activation function circuit 330, input memory 332, weight memory 334, output window 335, and control window 336. PE control signals 315 are output from the control window 336 to the PE circuits in the subclusters 338.”)
(Baum, “[0187] In one embodiment, the cluster supports multiple ANN layers in parallel, possibly from multiple ANNs. Note that a network layer can be implemented as a layer controller (LC) [to obtain a concurrent instruction] with one or more subclusters connected through the control interconnect, or one of the special units (special purpose, input or output) which contains the control within. Layers communicate data through the allocated memory blocks 298 in L3 memory 296, using signaling for flow control over the trigger interconnect, all defined by the configuration. The allocated memory blocks are also used as weight memory for the subclusters.”)
Baum teaches an input data set of the target network layer comprising a plurality of input tiles,
(Baum, “[0233] A diagram illustrating the flexible processing granularity of the NN processor and related memory versus latency trade-off is shown in FIG. 23. The data pipeline example, generally referenced 930, highlights the option of leveraging the data pipeline to favor minimal latency and operate at low input domain granularity. Consider the example input tensor 932 including input data 938 that can be located at the beginning of or at any arbitrary point in the network. One of the network layers then applies an NN operation 934 to the input data (e.g., 3×3 convolution in this example) [an input data set of the target network layer comprising a plurality of input tiles] followed by the output domain 936 including memory blocks 931 and 939.
[0234] In this example, the input data stream is fully consumed and all needed calculations are applied while minimizing latency and without the need to retrieve the input data since all computations are committed to intermediate results stored in memory. In alternative embodiments, this function can be executed by: (1) waiting for the entire frame and applying a batch operation whereby all data is immediately committed to output to avoid intermediate results; (2) waiting for the minimal set of rows in order to avoid intermediate results (in this example case three); (3) using intermediate results stored in external memory with the increase in memory access latency; or (4) recalling inputs as needed (i.e. multiple reads of the same data) in order to avoid having to store intermediate results.”)
PNG
media_image2.png
474
654
media_image2.png
Greyscale
Baum teaches and a depth of an input tile being obtained by performing adaptation processing according to a second quantity,
Examiner interprets adaption processing in light of the specification, ([0039], “The data adaption instruction instructs to adapt input data at any network layer to an input tile that matches the size of the on-chip cache of the processing chip.”)
(Baum, [0116], “The subcluster, in turn, comprises the most basic units, namely the processing elements (PEs) 76 which are composed of a multiply and accumulate (MAC) circuit and local memory.”)
(Baum, “[0150] The basic compute unit is a PE and comprises a multiply/accumulate entity that reflects the intrinsic operation of a neuron. The intermediate result or outcome is stored in L1 memory 150 which is local to the PE. The L1 memory has a certain depth and width, e.g., number of neurons P=16 [a depth of an input tile being obtained by performing adaptation processing according to a second quantity], each of which is 16 bits wide, in the example described herein. It is appreciated that L1 memory having any desired depth and width may be used. The depth P of L1 memory reflects the number of simultaneous ‘neurons’ or ‘contexts’ a PE can handle.”)
Baum teaches the computing engine comprising a plurality of processing elements
(Baum, “[0101] A high-level block diagram illustrating an example NN processing core in more detail is shown in FIG. 5. The NN processing engine or core 60 [computing engine] comprises several hierarchical computation units. The lowest hierarchical level is the processing element (PE) 76 with its own dedicated internal Layer 1 or L1 memory 78 in which individual neurons are implemented. A plurality of N PEs 76 along with dedicated Layer 2 or L2 memory 74 make up the next hierarchical level termed a subcluster 70 [comprising a plurality of processing elements].”)
Baum teaches and being configured to perform parallel processing on a target input tile in the input data set according to the concurrent instruction, to obtain target output data corresponding to the target input tile,
(Baum, “[0187] In one embodiment, the cluster supports multiple ANN layers in parallel [being configured to perform parallel processing on a target input tile in the input data set (recall fig. 23 for CNN tile) according to the concurrent instruction, to obtain target output data corresponding to the target input tile wherein the layer controller is a unit in the processing core ie the computing engine], possibly from multiple ANNs. Note that a network layer can be implemented as a layer controller (LC) with one or more subclusters connected through the control interconnect, or one of the special units (special purpose, input or output) which contains the control within.”)
Baum teaches wherein each time the computing engine performs the parallel processing, a first quantity of the plurality of processing elements are invoked simultaneously,
(Baum, “[0115] A third feature is the structure of the memory fabric including memory windowing. In addition to the localization and hierarchical structure of the memory, high bandwidth access to the memory is provided in parallel to a large number of computation units. This is achieved by narrowing access for a particular computation unit to only a small portion of the memory. Thus, full random access to the entire memory is not provided. Rather, access to only a relatively small window of memory is provided. This allows simultaneous access across thousands of computation units, thus representing a tradeoff between bandwidth and random accessibility. Since a single compute unit memory access pattern is structured and well-defined by the ANN and does not require full random access to the entire memory, access can be ‘windowed’ to only those few memory blocks required for that particular compute unit. Thus, extremely high memory bandwidth is achieved whereby thousands of compute units can access memory simultaneously in parallel with the tradeoff being access only to memory that is ‘local’ to the compute unit [wherein each time the computing engine performs the parallel processing, a first quantity of the plurality of processing elements are invoked simultaneously; wherein Examiner interprets each processing element to process a block as taught by Kuo and executed simultaneously].”)
Baum teaches the group cache unit being provided with the first quantity of output caches;
(Baum, “[0167] A high-level block diagram illustrating a first example subcluster in more detail is shown in FIG. 8. The subcluster, generally referenced 180, comprises a plurality of N PEs 182, each individual PE 182 including local L1 memory 184 [the group cache unit being provided with the first quantity of output caches; ie N number of L1 memory], interconnect fabric 186, dedicated local L2 memory 188 portioned into a plurality of allocated memory blocks 190, configuration and decode block 192, and control/data signals 181.”)
PNG
media_image3.png
522
663
media_image3.png
Greyscale
Baum teaches the group control unit being configured to store, by group, the target output data
(Baum, “[0151] The capability of handling internal context provides for a number of capabilities such as: (1) the ability to assign multiple logical neurons to a single physical neuron (each context stores the output of one neuron); (2) storing multiple intermediate results for the same input resulting in simultaneous operations [the group control unit being configured to store, by group, the target output data], and hypothesis testing for different versions of weights (e.g., backpropagation results, correction values based on gradients, etc.); (3) multithreaded inference of the same inputs for the purpose of applying common methodology of a network committee and a majority vote extraction; (4) running multiple networks if resources are available; and (5) load balancing based on overall network capacity as governed by an NN manager.”)
Baum teaches obtained by each of the first quantity of the plurality of processing elements processing the data with the depth of the second quantity,
(Baum, [0150], “The intermediate result or outcome is stored in L1 memory 150 which is local to the PE. The L1 memory has a certain depth and width, e.g., number of neurons P=16, each of which is 16 bits wide, in the example described herein. It is appreciated that L1 memory having any desired depth and width may be used. The depth P of L1 memory reflects the number of simultaneous ‘neurons’ or ‘contexts’ a PE can handle [obtained by each of the first quantity of the plurality of processing elements processing the data with the depth of the second quantity].”)
Baum teaches into a corresponding one of the first quantity of output caches of the group cache unit, wherein a same output cache is reused to store the target output data obtained in the depth direction by a corresponding processing element each time the parallel processing is performed.
(Baum, “[0167] A high-level block diagram illustrating a first example subcluster in more detail is shown in FIG. 8. The subcluster, generally referenced 180, comprises a plurality of N PEs 182, each individual PE 182 including local L1 memory 184, interconnect fabric 186, dedicated local L2 memory 188 portioned into a plurality of allocated memory blocks 190 [into a corresponding one of the first quantity of output caches ie L2 Memory of the group cache unit], configuration and decode block 192, and control/data signals 181. The configuration/decode circuit 192 receives instructions from an external control bus 194. Each subcluster 180 also communicates with input/output alignment circuit 196 and activation circuit 198 which in the example embodiment presented herein are located in the cluster hierarchy level, as described in more detail infra.
[0168] In one embodiment, the function of the subcluster is to aggregate a plurality of N PEs, e.g., N=64 [wherein a same output cache is reused to store the target output data obtained in the depth direction by a corresponding processing element each time the parallel processing is performed]. All PEs in a subcluster belong to the same layer of a neural network which greatly simplifies the control logic required. For example, apart from a static configuration a priori, control of cycle-by-cycle operation is not needed.”)
However, Baum does not explicitly teach the second quantity being an amount of processing data in each sliding window in a depth direction each time parallel processing is performed, and the adaptation processing being such that, in the depth direction, the input data set is adapted to the plurality of input tiles in which the second quantity of pieces of data are in one group; such that the artificial intelligence accelerator has a capability to respectively process data with a depth of the second quantity in parallel by simultaneously performing the first quantity of operation functions;
Kuo teaches the second quantity being an amount of processing data in each sliding window in a depth direction each time parallel processing is performed,
(Kuo, “[0031] In one embodiment, the input tiles may overlap with each other, and each tile is divided into equal-sized, non-overlapping blocks. A block (e.g., block 211) is a basic unit of computation [the second quantity being an amount of processing data in each sliding window in a depth direction each time parallel processing is performed; see Block 211 wherein the depth is denoted by C]. For example, an engine (e.g., the convolution engine 111) may include an array of multiply-and-accumulate (MAC) circuits, and the size of a block may be equal to the size of the MAC array. Thus, operations on a block can be performed in parallel within an engine. The size of an input tile may be determined by the size of the buffer (e.g., the convolution buffer 151). For example, an entire input tile should fit into the convolution buffer 151. In one embodiment, a programmer may run a compiler at design time to determine a number of tile sizes based on the available hardware components and expected characteristics of software applications to be run on the DLA 100.”)
PNG
media_image4.png
487
656
media_image4.png
Greyscale
Kuo teaches and the adaptation processing being such that, in the depth direction, the input data set is adapted to the plurality of input tiles in which the second quantity of pieces of data are in one group; such that the artificial intelligence accelerator has a capability to respectively process data with a depth of the second quantity in parallel by simultaneously performing the first quantity of operation functions;
(Kuo, “[0031] In one embodiment, the input tiles may overlap with each other, and each tile is divided into equal-sized, non-overlapping blocks. A block (e.g., block 211) is a basic unit of computation. For example, an engine (e.g., the convolution engine 111) may include an array of multiply-and-accumulate (MAC) circuits, and the size of a block may be equal to the size of the MAC array [the adaptation processing being such that, in the depth direction, the input data set is adapted to the plurality of input tiles in which the second quantity of pieces of data are in one group; wherein the multiply/accumulate entity (reflects the intrinsic operations of a neuron) of Baum is interpreted to be analogous to the MAC circuit of Kuo]. Thus, operations on a block can be performed in parallel within an engine [such that the artificial intelligence accelerator has a capability to respectively process data with a depth of the second quantity in parallel by simultaneously performing the first quantity of operation functions]. The size of an input tile may be determined by the size of the buffer (e.g., the convolution buffer 151). For example, an entire input tile should fit into the convolution buffer 151. In one embodiment, a programmer may run a compiler at design time to determine a number of tile sizes based on the available hardware components and expected characteristics of software applications to be run on the DLA 100.”)
Baum and Kuo are both considered to be analogous to the claimed invention because they are in the same field of neural network accelerators. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Baum to incorporate the teachings of Kuo in order to provide a standard convolution across multiple channels in a manner that can be performed in parallel utilizing an engine as doing so would allow parallelism to meet the need for increased system performance (Kuo, “[0004] Neural networks are not only computation-intensive, but also incur heavy memory access. For example, a core computation of neural network computing is convolution. For feature extraction, an input image may be convolved with a set of filters over a set of input channels (e.g., red, green and blue), followed by nonlinear computations, down-sampling computations, and class scores computations. The computations typically incur heavy data access to a system memory external to the processors. Thus, there is a need for improvement in neural network computing to increase system performance.”)
In regards to claim 2,
Baum and Kuo teaches The artificial intelligence accelerator according to claim 1,
Baum teaches wherein the control unit ie layer controller 82 is further configured to parse the processing instruction for the target network layer to obtain a migration instruction;
(Baum, [0117], “In addition, the NN processor includes layer controllers incorporating microcode machines that allow full accessibility to the control signaling of the computational elements, memory [parse the processing instruction for the target network layer to obtain a migration instruction] etc…
[0213] …The LCU itself is controlled by the contents of the instruction memory 654. The execution of each instruction results in the generation of encoded control signals which are then decoded by the decoders and output to the computing elements via the control window circuit 674. Note that in addition to the control signals that control the computing elements in the subclusters, the LCU also generates the control signals (i.e. MWC select controls) for controlling the control window as well (along with the weight, ingress and egress data windows). Once configured (as compile time), the control signals, weights, ingress and egress data are routed statically. The MMU 660 generates the control signals for the L3 memory windowing and functions to perform the virtual to physical mapping. It also functions to generate a contention alert 694 in response to a memory contention event between two layers in the ANN. As described supra, the LCU resolves the contention event by issuing one of the layers a halt command.”)
Baum teaches and the artificial intelligence accelerator further comprises: a full storage unit ie L3 memory (shared), configured to store the input data set of the target network layer and an output data set of the target network layer, the output data set comprising output data respectively corresponding to the plurality of input tiles; and
(Baum, “[0197] A diagram illustrating a second example memory windowing scheme is shown in FIG. 14. In one embodiment, the data that flows through the computing elements in the NN processor is pipelined, wherein PEs in the subclusters receive data as input and generate outputs which then serve as input for some other subcluster for subsequent computations…
[0198] The window memory scheme, generally referenced 340, comprises a plurality of subclusters 348, each including a plurality of PEs 349, L3 memory (not shared) 342, and L3 memory (shared) 344. In operation, the subclusters receive weights information 345 from a portion of L3 memory that is not shared. Input data 341 to a subcluster is received from an allocated memory block 346 from a shared portion of L3 memory. The PEs within the subcluster process the weights and input data and generate outputs 343. The outputs, however, are written to a different (e.g., neighboring) allocated memory block (i.e. not the memory block the inputs were read from). These outputs are then read as inputs to another subcluster (e.g., neurons in a subsequent layer of the ANN). In this fashion, ANN input data 347 enters shared L3 memory, is read from allocated memory blocks, processed by the PEs in one or more subclusters, output to neighboring memory blocks, and after traversing through the various layers in the ANN is ultimately output as ANN output data 349 from shared L3 memory [configured to store the input data set of the target network layer and an output data set of the target network layer, the output data set comprising output data respectively corresponding to the plurality of input tiles].”
PNG
media_image5.png
640
1052
media_image5.png
Greyscale
)
Baum teaches a migration engine ie layer controller, configured to perform a data migration operation between the full storage unit and the group cache unit according to the migration instruction.
(Baum, “[0210] A high-level block diagram illustrating the layer controller [a migration engine] interface to L3 memory and subclusters [configured to perform a data migration operation between the full storage unit and the group cache unit according to the migration instruction; see fig. 20] in more detail is shown in FIG. 20.”
PNG
media_image6.png
459
669
media_image6.png
Greyscale
)
In regards to claim 3,
Baum and Kuo teaches The artificial intelligence accelerator according to claim 2,
Baum teaches wherein the migration instruction comprises at least one of a load migration instruction or a store migration instruction; and the migration engine receives a load migration instruction from the control unit, and migrates an input tile in the full storage unit to the group cache unit according to the load migration instruction; and/or the migration engine receives a store migration instruction from the control unit, and migrates, according to the store migration instruction, output data cached in the group cache unit to the full storage unit.
(Baum, “[0197] A diagram illustrating a second example memory windowing scheme is shown in FIG. 14. In one embodiment, the data that flows through the computing elements in the NN processor is pipelined, wherein PEs in the subclusters receive data as input and generate outputs which then serve as input for some other subcluster for subsequent computations…
[0198] The window memory scheme, generally referenced 340, comprises a plurality of subclusters 348, each including a plurality of PEs 349, L3 memory (not shared) 342, and L3 memory (shared) 344. In operation, the subclusters receive weights information 345 from a portion of L3 memory that is not shared. Input data 341 to a subcluster is received from an allocated memory block 346 from a shared portion of L3 memory [wherein the migration instruction comprises at least one of a load migration instruction]. The PEs within the subcluster process the weights and input data [and the migration engine receives a load migration instruction from the control unit, and migrates an input tile in the full storage unit to the group cache unit according to the load migration instruction; wherein the subcluster receives input from the L3 memory (shared)] and generate outputs 343. The outputs, however, are written to a different (e.g., neighboring) allocated memory block (i.e. not the memory block the inputs were read from) [or a store migration instruction]. These outputs are then read as inputs to another subcluster (e.g., neurons in a subsequent layer of the ANN). In this fashion, ANN input data 347 enters shared L3 memory, is read from allocated memory blocks, processed by the PEs in one or more subclusters, output to neighboring memory blocks, and after traversing through the various layers in the ANN is ultimately output as ANN output data 349 from shared L3 memory.”
PNG
media_image5.png
640
1052
media_image5.png
Greyscale
)
In regards to claim 4,
Baum and Kuo teaches The artificial intelligence accelerator according to claim 3,
Baum teaches wherein the target network layer has a first operation parallelism degree and a second operation parallelism degree, the first operation parallelism degree indicating a quantity of operation functions comprised in the target network layer, and
(Baum, “[0150] The basic compute unit is a PE and comprises a multiply/accumulate entity that reflects the intrinsic operation of a neuron. The intermediate result or outcome is stored in L1 memory 150 which is local to the PE. The L1 memory has a certain depth and width, e.g., number of neurons P=16 [the first operation parallelism degree indicating a quantity
PNG
media_image7.png
515
81
media_image7.png
Greyscale
of operation functions comprised in the target network layer; wherein in this case, Examiner interprets the number of neurons to be equivalent to the quantity of operation functions in the network layer (see cropped figure 28 wherein Examiner notes the neurons of the layer (thus, operation functions) fits into one subcluster (subcluster 9))], each of which is 16 bits wide, in the example described herein. It is appreciated that L1 memory having any desired depth and width may be used. The depth P of L1 memory reflects the number of simultaneous ‘neurons’ or ‘contexts’ a PE can handle.”)
However, Baum does not explicitly teach the second operation parallelism degree indicating a processing data amount in a depth direction each time the target network layer performs parallel processing, the second operation parallelism degree being equal to the first quantity
Kuo teaches the second operation parallelism degree indicating a processing data amount in a depth direction each time the target network layer performs parallel processing, the second operation parallelism degree being equal to the first quantity.
(Kuo, “[0031] In one embodiment, the input tiles may overlap with each other, and each tile is divided into equal-sized, non-overlapping blocks. A block (e.g., block 211) is a basic unit of computation. For example, an engine (e.g., the convolution engine 111) may include an array of multiply-and-accumulate (MAC) circuits, and the size of a block may be equal to the size of the MAC array [the second operation parallelism degree indicating a processing data amount in a depth direction each time the target network layer performs parallel processing, the second operation parallelism degree being equal to the first quantity; wherein the size of the MAC array is interpreted to be analogous to the N number of processing elements as a processing element of Baum is composed of a MAC circuit and local memory].”)
(Baum, [0116], “The subcluster, in turn, comprises the most basic units, namely the processing elements (PEs) 76 which are composed of a multiply and accumulate (MAC) circuit and local memory.”)
In regards to claim 5,
Baum and Kuo teaches The artificial intelligence accelerator according to claim 4,
PNG
media_image8.png
504
76
media_image8.png
Greyscale
Baum teaches wherein the first operation parallelism degree is greater than the first quantity, and the computing engine groups operation functions in the target network layer into P function groups, …, P being determined according to a ratio of M to N, M representing the first operation parallelism degree ie number of operations in the layer and N representing the first quantity ie number of processing elements , M and N being positive integers.
(Baum, Fig. 28 [wherein the first operation parallelism degree is greater than the first quantity; see fig. 28 wherein the total number of operations in a layer is greater than the number of PEs in a subcluster] and Table 1, “[0139] Where N represents the number of processing elements in a subcluster, M is the number of subclusters in a cluster [the computing engine groups operation functions in the target network layer into P function groups (see cropped fig. 28 wherein the network layer 1 neurons are divided into three subclusters)… P being determined according to a ratio of M to N… M representing the first operation parallelism degree ie number of operations in the layer and N representing the first quantity ie number of processing elements], and L is the number of clusters in the NN processor device. Note that the size indicated for each memory level L1 through L5 are for illustration purposes only. It is appreciated that any desired memory size for the various memory layers may be implemented without departing from the scope of the invention.”)
PNG
media_image9.png
303
619
media_image9.png
Greyscale
Baum teaches successively invokes operation functions in each function group according to the concurrent instruction to perform parallel processing on the target input tile
(Baum, “[0197] A diagram illustrating a second example memory windowing scheme is shown in FIG. 14. In one embodiment, the data that flows through the computing elements in the NN processor is pipelined, wherein PEs in the subclusters receive data as input and generate outputs which then serve as input for some other subcluster for subsequent computations…
[0198] The window memory scheme, generally referenced 340, comprises a plurality of subclusters 348, each including a plurality of PEs 349, L3 memory (not shared) 342, and L3 memory (shared) 344. In operation, the subclusters receive weights information 345 from a portion of L3 memory that is not shared. Input data 341 to a subcluster is received from an allocated memory block 346 from a shared portion of L3 memory. The PEs within the subcluster process the weights and input data and generate outputs 343. The outputs, however, are written to a different (e.g., neighboring) allocated memory block (i.e. not the memory block the inputs were read from). These outputs are then read as inputs to another subcluster (e.g., neurons in a subsequent layer of the ANN) [successively invokes operation functions in each function group according to the concurrent instruction to perform parallel processing on the target input tile; ie wherein each function group is a subcluster and the PEs are executed concurrently]. In this fashion, ANN input data 347 enters shared L3 memory, is read from allocated memory blocks, processed by the PEs in one or more subclusters, output to neighboring memory blocks, and after traversing through the various layers in the ANN is ultimately output as ANN output data 349 from shared L3 memory.”
PNG
media_image5.png
640
1052
media_image5.png
Greyscale
)
In regards to claim 6,
Baum and Kuo teaches The artificial intelligence accelerator according to claim 5,
PNG
media_image10.png
615
474
media_image10.png
Greyscale
Baum teaches wherein the group cache unit ie subcluster 70 comprises an input cache ie L1 memory,
(Baum, “[0101] A high-level block diagram illustrating an example NN processing core in more detail is shown in FIG. 5. The NN processing engine or core 60 comprises several hierarchical computation units. The lowest hierarchical level is the processing element (PE) 76 with its own dedicated internal Layer 1 or L1 memory 78 in which individual neurons are implemented. A plurality of N PEs 76 along with dedicated Layer 2 or L2 memory 74 make up the next hierarchical level termed a subcluster 70 [wherein the group cache unit ie subcluster 70 comprises an input cache ie L1 memory].”)
Baum teaches and the target input tile independently uses one load migration instruction, one concurrent instruction, and one store migration instruction;
(Baum, “[0144] In one embodiment, the basic compute unit is the processing element (PE). A block diagram illustrating an example low-level processing element (PE) in more detail is shown in FIG. 6. The PE, generally referenced 140, comprises one or more multipliers 142 controlled by multiply trigger 177, an adder 144 controlled by adder trigger 171, L1 memory 150 comprising a plurality of registers 152, destination multiplexer 146 controlled by destination control 175, source multiplexer 148 controlled by source control 173, write multiplexer 154 controlled by output shuffle control 178, and read multiplexer 156 controlled by input shuffle control 179 [and the target input tile independently uses one load migration instruction, one concurrent instruction, and one store migration instruction].”)
Baum teaches the migration engine migrates the target input tile from the full storage unit ie L3 memory (shared) to the input cache ie L1 memory of the group cache unit according to a load migration instruction corresponding to the target input tile;
(Baum, See figure 6 166 From L2/L3 memory)
Baum teaches the computing engine ie processing core reads the target input tile from the input cache ie L1 memory by using the group control unit ie NN manager, and performs parallel processing on the target input tile according to a concurrent instruction corresponding to the target input tile, to obtain the target output data corresponding to the target input tile;
(Baum, “[0103] In one embodiment, the NN manager 62 is a specialized processor that controls two data pipes: one parallel and one serial along with functions to drive the network fabric. This processor carries out special purpose operations that are native to the control plane of the neural network [the computing engine ie processing core reads the target input tile from the input cache ie L1 memory by using the group control unit ie NN manager, and performs parallel processing on the target input tile according to a concurrent instruction corresponding to the target input tile, to obtain the target output data corresponding to the target input tile]. Example operations includes, but are not limited to, Infer, Train, Load weights, and Update weights. Load balancing and resource allocation are handled by an external software tool chain, which includes a set of tools including a compiler, mapper, and allocator, that address these tasks.”)
Baum teaches and the migration engine ie layer controller migrates the target output data in the corresponding one of the first quantity of output caches to the full storage unit ie L3 memory (shared) according to a store migration instruction corresponding to the target input tile.
(Baum, “[0214] A high-level block diagram illustrating a second example layer controller in more detail is shown in FIG. 21. The example LC, generally referenced 550, comprises instruction memory 552 including a plurality of instructions 554, LCU 556, instruction decoders 566, trigger window crossconnect 558, and trigger handler 560. The LCU 556 comprises a state machine 562, and instruction register 564 [according to a store migration instruction corresponding to the target input tile; see fig. 21].
[0215] In operation, instructions 551 are read from instruction memory into the instructions register 564 in the LCU where they are decided and executed. The one or more portions 568 of the instruction that are configured to directly control hardware are sent to the one or more decoders 566 for decoding. The output of the decoders comprises direct control signaling that is sent to the subclusters to control the internal PE operation as shown and described supra in FIG. 20 [migrates the target output data in the corresponding one of the first quantity of output caches to the full storage unit; see fig. 20]. The other portions 570, 572 of the instruction control the logical state of the LCU and are input to the state machine 562. These portions control looping and branching, for example. A next 553 command causes the next instruction from the instruction memory 552 to be read into the LCU for execution.
PNG
media_image11.png
500
651
media_image11.png
Greyscale
PNG
media_image12.png
463
668
media_image12.png
Greyscale
)
In regards to claim 7,
Baum and Kuo teaches The artificial intelligence accelerator according to claim 5,
Baum teaches wherein a quantity of the target output data is M, the M pieces of target output data are grouped into P groups, and each group comprises N pieces of target output data; and the group control unit stores an nth piece of target output data in each group into an nth output cache of the group cache unit, n E[1, N].
(Baum, Fig. 28 [wherein a quantity of the target output data is M; see fig. 28 for a total number of operations in a layer] and Table 1, “[0139] Where N represents the number of processing elements in a subcluster [and the group control unit stores an nth piece of target output data in each group into an nth output cache ie nth L1 memory ie respective L1 memory of the PE of the group cache unit, n E[1, N]], M is the number of subclusters in a cluster [the M pieces of target output data are grouped into P groups ie number of operations in the layer and each group comprises N ie number of processing elements pieces of target output data], and L is the number of clusters in the NN processor device. Note that the size indicated for each memory level L1 through L5 are for illustration purposes only. It is appreciated that any desired memory size for the various memory layers may be implemented without departing from the scope of the invention.”)
Wherein Baum discloses the depth of P can be any desired depth and width
(Baum, [0150] “The L1 memory has a certain depth and width, e.g., number of neurons P=16, each of which is 16 bits wide, in the example described herein. It is appreciated that L1 memory having any desired depth and width may be used. The depth P of L1 memory reflects the number of simultaneous ‘neurons’ or ‘contexts’ a PE can handle.”)
Claim 13 is rejected on the same grounds under 35 U.S.C. 103 as analogous claim 1 and as they are substantially similar.
Claim 14 is rejected on the same grounds under 35 U.S.C. 103 as analogous claim 2 as they are substantially similar.
Claim 15 is rejected on the same grounds under 35 U.S.C. 103 as analogous claim 3 as they are substantially similar.
Claim 16 is rejected on the same grounds under 35 U.S.C. 103 as analogous claim 4 as they are substantially similar.
Claim 17 is rejected on the same grounds under 35 U.S.C. 103 as analogous claim 5 as they are substantially similar.
Claim 18 is rejected on the same grounds under 35 U.S.C. 103 as analogous claim 6 as they are substantially similar.
Claims 19 and 20 are rejected on the same grounds under 35 U.S.C. 103 as analogous claim 1 as they are substantially similar.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baum in view of Kuo in further view of U.S. Pub. No. US20190050717A1 Temam et al. (“Temam”)
In regards to claim 12,
Baum and Kuo teaches The artificial intelligence accelerator according to claim 2,
Temam teaches further comprising: an instruction generation unit, configured to: generate a processing instruction for each network layer in the neural network model, and store the processing instruction for each network layer into the full storage unit;
(Temam, “[0035] As depicted, host interface 108 is coupled to I/O link 110, controller 102, and classifier portion 116. Host interface 108 receives instructions and data parameters from I/O link 110 and provides instructions and parameters [an instruction generation unit, configured to: generate a processing instruction for each network layer in the neural network model; wherein no steps are recited for generating a processing instruction thus receiving and sending both the instruction and the parameters suffices as generating a processing instruction] to controller 102. In general, instructions can be provided to one or more devices in system 100 through instruction bus 124 (described below) and parameters can be provided to one or more devices in system 100 through ring bus 128 (described below). In some implementations, instructions are received by controller 102 from host interface 118 at an initial time and stored in instruction memory 106 [store the processing instruction for each network layer into the full storage unit] for execution by controller 102 at a later time.”)
Temam teaches and load the processing instruction for the target network layer of the neural network model from the full storage unit, and cache the processing instruction for the target network layer for reading by the control unit.
(Temam, “[0099] A second embodiment includes preloading/caching the same subset of the (currently used) parameters in all tiles [load the processing instruction for the target network layer of the neural network model from the full storage unit, and cache the processing instruction for the target network layer for reading by the control unit], as the tiles use the same parameters at the same time. During execution, the parameters (subset), not the partial activations sums, rotate around the ring.”)
Temam is considered to be analogous to the claimed invention because they are in the same field of neural network accelerators. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Baum and Kuo to incorporate the teachings of Temam in order to provide a ring bus to provide sufficient bandwidth to load parameters (Temam, “[0096] Convolutional neural networks pass parameters from one tile to another. For neural networks largely composed of convolution layers, where parameters are reused across neurons (also known as activations), the memory bandwidth requirements are less high, but still usually higher than a typical external memory. The ring bandwidth can be sufficient to load the parameters to the tile, provided it is connected to a large on-die memory of the same width as the ring.”)
Allowable Subject Matter
Claims 8-11 were searched but not rejected under 35 USC §102 and §103.
In addition, examiner notes, the noted claims should also be amended to overcome the claim rejections indicated in the current office action; wherein the amendments do not raise new issues that would require an updated rejection of claims.
Claims 8-11 contain allowable subject matter.
The following is an examiner’s statement of reasons for allowance:
The references of record alone or in combination do not disclose or suggest the limitations found within claims 8-11 limitations as a whole with regards to technical features recited by the claim limitations directed to:
“the target filling function group comprises M operation functions and (N-M) filling functions in the target network layer, N function bits are set in the target filling function group, and a value range of the N function bits is [0, N-1];
the M operation functions are set in valid function bits in the N function bits, and the (N-M) filling functions are set in filling function bits in the N function bits; and a value range of a valid function bit is [(i-1)*M, i*M-1], and a filling function bit is a function bit other than a valid bit in the N function bits;
and i represents the arrangement position of the target input tile in the target input data group, i E [1, I]” (in exemplar claim 8);
In regards to claim 8, Baum and Kuo teaches The artificial intelligence accelerator according to claim 4,
Cheltur teaches wherein the first operation parallelism degree ie number of threads in a thread group (analogous to the number of neurons in Baum) is less than the first quantity ie number of filters K (number of processing elements in Baum),
(Cheltur, “[0047] In operation, each SM 310 is configured to process one or more thread groups. As used herein, a “thread group” or “warp” refers to a group of threads concurrently executing the same program on different input data, with one thread of the group being assigned to a different execution unit within an SM 310. A thread group may include fewer threads [first operation parallelism degree is less than] than the number of execution units [the first quantity; wherein the number of execution units perform the operations, which is represented by the number of filters K] within the SM 310, in which case some of the execution may be idle during cycles when that thread group is being processed.”)
(Cheltur, “[0071] FIG. 6 illustrates the streaming multiprocessor 310 of FIG. 3 configured to perform a multi-convolution operation, according to one embodiment of the present invention. In the context of FIG. 4, the convolution engine 125 configures functional units (e.g., execution units, load-store units, etc.) included in the streaming multiprocessor (SM) 310 to perform operations that implement multi-convolution operations. For explanatory purposes, operations performed by the SM 310, including the functional execution units, that are configured by the convolution engine 125 are also referred to herein as operations performed by the convolution engine 125 [the first quantity; wherein the number of operations is represented by the number of filters].”)
Cheltur teaches and the group control unit groups the input tiles in the input data set into a plurality of input data groups ie cells of the virtual image matrix, each input data group comprising I ie number of cells in the virtual image matrix successively arranged input tiles, and I being determined according to a ratio of N to M;
(Cheltur, fig. 5, “[0063] As part of the conversion between the image batch 410 and the virtual image matrix 510, each of the rows [plurality of input data groups; each input data group comprising I successively arranged input tiles] of the virtual image matrix 510 is associated with the values included in the image batch 410 that are required to compute one or more of the output images 480 included in the output batch 470. Such a conversion includes duplication of some of the values included in the image batch 410. For example, as depicted for the value “D4,” the center of each of the three-by-three color planes 410 is used four times [N] to compute each of four feature maps 490 [M] and, consequently, each of the center values (e.g., the “D4” values) is associated with four separate rows of the virtual image matrix 510. As a result, multiple locations in the virtual image matrix 510 are associated with a single location in the image batch 410 [I being determined according to a ratio of N to M].”)
Cheltur in view of Qiu, Jiantao, et al. "Going deeper with embedded FPGA platform for convolutional neural network." teaches the computing engine performs offset filling on operation functions in the target network layer
(Qiu, Section 6.2, “Bias Shift module and Data Shift module are designed to support dynamic quantization. Input bias will be shifted by Bias Shift according to the layer’s quantization result. For a 16-bit implementation, the bias is extended to 32-bit to be added with convolution result [offset filling on operation functions in the target network layer].”)
However, Qiu does not explicitly teach according to an arrangement position of the target input tile in a target input data group to which the target input tile, to obtain a target filling function group;
U.S. Pub. No. US20190213005A1 Temam et al. (“Temam-07”) teaches a value range of [a valid function bit] is [(i-1)*M, i*M-1],
(Temam-07, “[0067] The following provides template parameters that may be used to instantiate a specialized TTU 300: 1) X Number of TTU Rows; 2) X Number of TTU Counters per Row; 3) X number of TTU Adder Units; 4) per TTU Row indicate shared Adder Reference; and 5) per Counter indicate X Counter Size [TTU][Row][Depth] [value range of a valid function bit is [(i-1)*M, i*M-1]; wherein the counter is i and the size is M and valid is interpreted to be within this range]. All TTU registers are architecturally visible. An address of a particular tensor element (i.e., tensor address 312) that needs to be accessed for the computation is the result of the addition of the counters. When an increment signal is issued from the control thread to a row of the TTU, TTU 300 executes a single cycle operation and increments an innermost dimension by a stride 304 of that dimension and propagates the rollover through all the depths.”)
Temam-07 further teaches the size in terms of bits/bytes
(Temam-07, [0035], “Wide and narrow designations are used throughout the specification and generally refer to an approximate size in width (bits/bytes) of one or more memory units. As used herein, “narrow” may refer to one or more memory units each having a size or width of less than 16-bits and “wide” may refer to one or more memory units each having a size or width or less than 64-bits.”)
However, Cheltur in view of Qiu and Temam-07 fail to teach “the target filling function group comprises M operation functions and (N-M) filling functions in the target network layer, N function bits are set in the target filling function group, and a value range of the N function bits is [0, N-1];
the M operation functions are set in valid function bits in the N function bits, and the (N-M) filling functions are set in filling function bits in the N function bits; and a value range of a valid function bit is [(i-1)*M, i*M-1], and a filling function bit is a function bit other than a valid bit in the N function bits;
and i represents the arrangement position of the target input tile in the target input data group, i E [1, I]” in claim 8.
Thus, dependent claims 9-11 are not taught.
In summary, the references made of record, fail to disclose the required claimed technical features recited by the noted claim limitations as a whole; and the related dependent claims are found allowable for the reasons noted above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US Pub No. US20180314671A1: Zhang et al. teaches Systems And Methods For Systolic Array Design From A High-Level Program
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASMINE THAI whose telephone number is (703)756-5904. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.T./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights