DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
(Submitted on 4/16/2026)
The examiner notes that the applicant has moved the claim 9 limitations to claim 1 and has subsequently CANCELLED claim 9. The applicant has added a new claim 20. The examiner presents based on the broadest reasonable interpretation that the invention as “ A hardware accelerator to process neural network operations using an “interleaved pipeline” in which the pipeline is a hardware pipeline of plurality of hardware units (processors) interconnected to each other via crossbar to map the neural network operations to these units to provide an optimized performance”. In this context, the new reference “BOESCH” teaches a specific reconfigurable accelerator fabric for neural network operations consistent with the secondary reference “Ramesh” used in the earlier Office Action teaching an overarching reconfigurable fabric for all application processing acceleration engines that may include the neural network operations[See Col 2, lines 55-60].
In regard to 101 Rejections
The applicant has amended to explicitly state a “hardware pipeline” (a sequence of the hardware units interconnected using an interconnection fabric” with the focus of layer processing in a accelerator fabric which is detailed from Pages 8-11. The applicant argues on Page 12 and states the entire point and purpose of the invention is to configure a neural network accelerator crossbar to form a hardware pipeline in a determined order for processing input data.
Examiner’s Response:
The examiner submits that the applicant argument are persuasive and hereby WITHDRAWS the 101 rejections on claims 1-8, 10-17 and 19. As claim 20 is a new claim dependent on Claim 1, the claim 20 does not have any 101 rejection implications.
In regard to 103 Rejections
On Page 13, the applicant argues on reference Mody that the reference does not read memory with a predefines restrictions for connectivity of valid hardware units via the crossbar. The applicant argues from Page 15-16 on Ramesh reference which is used as a “Secondary Reference” that “Ramesh” does not teach a neural network.
Examiner’s Response
The examiner respectfully disagree with the applicant’s argument regarding reference “Ramesh”. Ramesh teaches cognitive brain mimicking a NN in which the morphing processing is a layer processing and state of morphing does represent “weight”. Just because the term NN not explicitly called in the Ramesh reference( used as a secondary reference) does not make the reference irrelevant to the domain of neural network. Without conceding the applicant arguments over “Mody” and “Ramesh” references, the examiner uses three new references “FEEHRER” , “BOESCH” and “Naga” to teach the amended claims. Combination of these three new references may provide strong teaching of the entire invention. The examiner interprets using BRI that the reference “Ramesh” as a secondary reference to “BOESCH” as primary reference may even teach the entire invention.
In CONCLUSION, the examine hereby rejects claims 1-8, 10-17 and 19 and rejects claim 20 under 103 and MOVE the application to FINAL REJECTION.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 11-12, 14-17 and 19-20 are rejected under 35 U.S.C. 103 unpatentable over
John FEEHRER et.al. (hereinafter FEEHRER) US 2021/0133122 A1,
In view of Thomas BOESCH et.al. (hereinafter BOESCH) US 2018/0189641 A1,
In view of Rahul Nagarajan et.al. (hereinafter Naga) US 2021/0303978 A1.
In regard to claim 1: (Currently Amended)
FEEHRER discloses:
- A computer-implemented method for configuring a neural network accelerator to process input data,
[0218]:
FIG. 23 is a conceptual diagram of a processing system 500 implemented using the GPU 102, in accordance with an embodiment. The exemplary system 500 may be configured to implement the methods disclosed in this application.
[0156]:
The GPU 102 may be configured to accelerate numerous deep learning systems and applications including autonomous vehicle platforms, deep learning,
[0168]:
In an embodiment, the operation of the GPC 350 is controlled by the pipeline manager 410. The pipeline manager 410 manages the configuration of the one or more DPCs 420 for processing tasks allocated to the GPC 350.
[0168]:
In an embodiment, the pipeline manager 410 may configure at least one of the one or more DPCs 420 to implement a neural network model and/or a computing pipeline.
- the neural network accelerator comprising a plurality of hardware processing units and a crossbar, the crossbar coupled to each hardware processing unit of the plurality of hardware processing units, each hardware processing unit comprising hardware to accelerate performing one or more neural network operations on received data,
[0157]:
As shown in FIG. 18, the GPU 102 includes an Input/Output (I/O) unit 305, a front end unit 315, a scheduler unit 320, a work distribution unit 325, a hub 330, a crossbar (Xbar) 370, one or more general processing clusters (GPCs) 350, and one or more partition units 380. The GPU 102 may be connected to a host processor or other PPUs 300 via one or more high-speed NVLINK™ 310 interconnects forming an interconnect fabric
PNG
media_image1.png
672
517
media_image1.png
Greyscale
[0167]:
FIG. 19 illustrates a GPC 350 of the GPU 102 of FIG. 18, in accordance with an embodiment. As shown in FIG. 19, each GPC 350 includes a number of hardware units for processing tasks. In an embodiment, each GPC 350 includes a pipeline manager 410, a pre-raster operations unit (PROP) 415, a raster engine 425, a work distribution crossbar (WDX) 480, a memory management unit (MMU) 490, and one or more Data Processing Clusters (DPCs) 420.
[0156]:
As discussed above, one or more GPUs 102 as shown may be configured to accelerate thousands of High Performance Computing (HPC), data center, and machine learning applications. The GPU 102 may be configured to accelerate numerous deep learning systems and applications
FEEHRER does not explicitly disclose:
- obtaining a sequence of one or more neural network operations to be performed by the neural network accelerator on the input data;
- selecting a set of hardware processing units from the plurality of hardware processing units to perform the one or more neural network operations
- reading, from a memory, a predefined set of restrictions defining which hardware processing units can be validly connected to each other using the crossbar;
- determining an order of the selected set of hardware processing units to perform the one or more neural network operations in accordance with the sequence;
- and providing the neural network accelerator with control information that causes the crossbar of the neural network accelerator to form a hardware pipeline of the selected set of hardware processing units in the determined order to process the input data so as to configure the neural network accelerator to process the input data via the hardware pipeline of the selected set of hardware processing units in the determined order.
However, BOESCH discloses:
- obtaining a sequence of one or more neural network operations to be performed by the neural network accelerator on the input data;
[0067]:
In an exemplary architecture, two or more (e.g., eight) digital signal processor (DSP) clusters are formed in a system on chip (SoC). Each DSP cluster may include two or more DSP's,
[0070] :
In the exemplary architecture a design time configurable accelerator framework (CAF) includes unidirectional links transporting data streams via a configurable, fully connected switch to, from, or to and from source devices and sink devices. The source and sink devices may include any one or more of DMA's, input/output (I/O) interfaces (e.g., multimedia, satellite, radar, etc.), and various types of accelerators including one or more convolution accelerators (CA).
[0142]:
Each DSP cluster 122, 140 includes a plurality (e.g., two) of DSPs 142, 152, a plurality (e.g., two) of local DSP crossbar switches 144, 154, and a DSP cluster crossbar switch 145, 155. Each DSP 142, 152 in a particular cluster communicates with each other via the DSP cluster crossbar switch 145, 155.
[0139]:
the applications processor 128 may coordinate the reconfiguration of the CAF 400 or DSPs 138 based on the configuration file, which itself may be based on the DCNN layers and topology.
[0271] :
in some cases, when two or more kernels are processed in parallel and the lines are not interleaved, data from the CA adder tree 622 is passed through the CA output interface 608 as a fixed sequence. That is, in the fixed sequence, the result for a first kernel may be passed through the CA output interface 608, then the result for a second kernel, and so on. In this case, the output data can be handled in a manner similar to the way that batch size is used for kernel data and input feature data. For example, if four (4) kernels are processed in parallel, the output feature data will be processed as if a batch size equals four (4). Correspondingly in these cases, intermediate data passed through the first CA input interface 602 will also maintain a same sequence.
[0074]:
The DSP's perform pooling (e.g., max pooling, average pooling, etc.), nonlinear activation, cross-channel response normalization, and classification representing a selected fraction of the total DCNN computation in an architecture that is flexible and amenable to future algorithmic evolutions [ BRI: A sequence of NN operations on a convolution accelerator typically includes convolution and activation, and in many cases they are fused into a single hardware stage for performance gains ].
- selecting a set of hardware processing units from the plurality of hardware processing units to perform the one or more neural network operations
[0152]:
The CAF 400 allows for the definition of a selectable number of concurrent, virtual processing chains at run time. The CAF 400 also includes a full featured back pressure mechanism to control data flow to the various components of the framework. The CAF 400 is arranged for stream multicasting operations, which enable the reuse of a data stream at multiple block instance
[0151]:
the CAF 400 is re-configured during run time based on defined DCNN layers and topology or information received from one or more DSPs 138, applications processor 128, or the like. [BRI: A virtual processing chain is a logical pipeline that can be instantiated from a set of physical processing units. With reconfigurable CAF, plurality set of hardware units can be selected]
[0198]:
The selection mechanism 508 is directed according to stream switch configuration logic 510. The stream switch configuration logic 510 determines at run time which input port 504 shall supply data to the associated output port, and based on the determination, the stream switch configuration logic 510 forms an appropriate selection signal that is passed to the data switch 506. The stream switch configuration logic 510 operates at run time and in real time. The stream switch 510 may take direction from CAF control registers, from a DSP of the DSP cluster 122 (FIG. 3), from the application processor 128, or from some other control device. In addition, the stream switch configuration logic 510 may also take direction from message/command logic 512.
- reading, from a memory, a predefined set of restrictions defining which hardware processing units can be validly connected to each other using the crossbar;
[0182]:
One or more convolution accelerator template modules may be included in an IP modules library such as the library described with respect to Table 2.
[0182]:
A predefined set of configuration registers can be extended. Configurable FIFOs formed or otherwise located at the stream link ports can be used to absorb data rate fluctuations and provide some buffering margin required to relax certain flow control constraints in a processing chain. [BRI: a processing chain is a “hardware pipeline”]
- determining an order of the selected set of hardware processing units to perform the one or more neural network operations in accordance with the sequence;
[0278:
Using the feature data, kernel data, and intermediate data, the CA 600 performs one or more sequences of serial or parallel convolution operations to produce output data.
[0169]:
The DMA engines 406 are arranged to provide bidirectional channels for input data flow, output data flow, or input and output data flow. In these cases, substantial quantities of data is passed into the CAF 400, out from the CAF 400, or into and out from the CAF 400
[0157]:
The CAF 400 includes a stream switch 500 that provides a design time parametric, run-time reconfigurable accelerator interconnect framework to support data-flow based processing chains. [BRI: in both serial and parallel convolution chains, the data flow defines a processing order. In serial chains, this is a strict sequential order; in parallel chains, it’s a combined order after merging branches]
PNG
media_image2.png
541
576
media_image2.png
Greyscale
[0164]:
Continuing in the description of FIG. 4, in addition to the stream switch 500, the CAF 400 may also include a system bus interface module 404. The system bus interface module 404 provides an interface to other modules of SoC 110. As shown in the exemplary embodiment of FIG. 3, the CAF 400 is coupled to the secondary communication bus 166. In other cases, the CAF 400 may be coupled to the primary communication bus 132 or some other communication mechanism. Control information may be passed unidirectionally or bidirectionally through the system bus interface module 404 of the CAF 400. Such interface is used to provide a host processor (e.g., DSP of DSP cluster 130, applications processor 128, or another processor) access to all of the CAF control registers 402,
[0144]:
Additionally, the global DSP cluster crossbar switch 150 is communicatively coupled to a system bus 166 (e.g., secondary communications bus, xbar—SoC crossbar switch, or the like), which enables each DSP to communicate with other components of the SoC 110
[0161]:
The stream switch 500 is a unidirectional interconnection structure formed with a plurality of unidirectional “stream links.” The stream links are arranged to transport multibit data streams from accelerators, interfaces, and other logic modules to the stream switch 500 and from the stream switch 500 to accelerators, interfaces, and other logic modules
- and providing the neural network accelerator with control information that causes the crossbar of the neural network accelerator to form a hardware pipeline of the selected set of hardware processing units in the determined order to process the input data so as to configure the neural network accelerator to process the input data via the hardware pipeline of the selected set of hardware processing units in the determined order.
[0157]:
The CAF 400 includes a stream switch 500 that provides a design time parametric, run-time reconfigurable accelerator interconnect framework to support data-flow based processing chains. [BRI: A processing chain is a “hardware pipeline”][0244]:
At the start of a line in first cycle, a first CA MAC unit 620 of each of three clusters (i.e., three 1.sup.st MAC units) performs calculations of the first column for the first output value of the first line. [BRI: CA is a convolution accelerator. This does exploit pipeline and parallelism for efficiency. The “determined order” implies that the data flow is synchronized across clusters. For example, the first cluster’s accelerator might process the first line, then pass it to the second cluster’s first accelerator, and so on. This combines line-based streaming, parallel cluster processing, and pipelined dataflow to maximize throughput and minimize idle cycles]
[0158]:
When integrating known hardware data path accelerators, the inventors have recognized that system designers must often choose between tightly coupled or loosely coupled architectures. Based on this choice, particular programmable elements (e.g., processors, DSPs, programmable logic controllers (PLC's), and other programmable devices) deployed in such a device (e.g., a system on chip) would be required to adopt a suitable, but limiting programming model.
[0158]:
This is undesirable and limiting, however, because the hardwired data paths provide little if any flexibility at run-time,
[0159]:
To overcome these limitations, a reconfigurable data transfer fabric of the stream switch 500
[0144]:
the global DSP cluster crossbar switch 150 is communicatively coupled to a system bus 166 (e.g., secondary communications bus, xbar—SoC crossbar switch, or the like), which enables each DSP to communicate with other components of the SoC 110. For example, each DSP 142, 152 can communicate with one or more components (e.g., one or more convolution accelerators) of the CAF 400 or access an SoC global memory 126 via the system bus 166.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
One of ordinary skill would have motivation to combine FEEHRER and BOESCH that can use an on-chip an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic ( BOESCH [0123]).
FEEHRER and BOESCH do not explicitly disclose:
- such that the restrictions are not contravened and only valid combinations of hardware processing units are to be connected using the crossbar [Determining the order]
However, Naga discloses:
- such that the restrictions are not contravened and only valid combinations of hardware processing units are to be connected using the crossbar [Determining the order]
[0006]:
The retrieved data can represent inputs to a neural network layer. Each of the requests is distributed with reference to a channel controller that is selected to process the request. The requests to retrieve the inputs are distributed to the channel controllers for processing in a manner that reduces or eliminates load imbalances across the channel controllers. In this example the retrieved data is processed to perform neural network computations. In some instances the data is processed as a step in accelerating computations of an embedding layer of an artificial neural network.
[0016]:
Circuitry for a crossbar/on-chip interconnect can be implemented at a special-purpose hardware circuit, such as a hardware accelerator used in a distributed system. The crossbar allows each channel controller to read data from, and write data to, any address location of a memory cell in any channel of a high-bandwidth memory system that communicates with a processor core or accelerator chip.
[0064] :
the modified round-robin dispatch algorithm 300 is configured to disrupt or inhibit latent pathological sequences that can occur during data accesses for a machine-learning workload. Because of this, the modified round-robin dispatch algorithm 300 is configured to allow allocations of ID headers (e.g., address of activations or gradients) in a manner that is load balanced across each channel controller 202 in a set of channel controllers (350). A standard round-robin approach for scheduling a process indicates to select a channel controller in a simple, circular order in which selections are performed without priority.
[0065]:
the round-robin approach can be adapted or modified to first detect an initial completion of a first circular order of selections. [BRI: in a simple, circular order without priority, where each selection is made in turn and all options are treated equally. Selections are made in a fixed circular sequence. Each unit gets a turn before the next is chosen, and then the scheduling discipline is round-robin]
[0056] :
prior approach of mapping specific channel controllers 202 to a particular memory channel can have other challenges. For example, the approach can have a constraint of requiring data be stored in a manner that is sensitive to how the addresses and data are mapped to specific channel controllers 202.
[0024] :
FIG. 4 illustrates an example allocation of requests to different channel controllers.
[0036]:
data processing techniques and corresponding hardware circuitry that can be implemented in a special-purpose processor to balance processing loads experienced by channel controllers in a distributed processing system. For example, a distributed system that includes a large memory unit (e.g., a high-bandwidth memory) and a special-purpose hardware circuit can generate instructions to cause any channel controller to obtain data from any memory location and for any data shard of the memory unit. More specifically, this feature is enabled based on an on-chip interconnect (or crossbar) that is integrated at the hardware circuit to allow each channel controller to read data from, and write data to, any channel of a high-bandwidth memory system. The crossbar feature removes the constraint of storing data in a manner that is sensitive to which addresses allocations are mapped to specific channel controllers and allows for simplifying how sets of data may be laid out in the memory system.
[0037]:
the system includes an example hardware manager that executes instructions for defining and managing the each circular buffer. Instead of allocating a fixed size amount of shared memory buffers to each channel controller, the hardware manager is operable to define a size of each buffer allocation based on an observed latency required to fully execute computes on data fetched from memory locations of the system memory.
[BRI: the need for load balancing in channel controllers must be designed so that it does not violate the crossbar’s hardware constraints. The requires only connecting valid combinations of processing and routing decisions should be made within those constraints to maintain system stability, performance, and scalability
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, BOESCH and Naga.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches restrictions are not contravened when an order for the selected hardware units are determined.
One of ordinary skill would have motivation to combine FEEHRER, BOESCH and NAGA that can provide improve efficiency for neural network operations particularly load balancing( Naga[0006])
In regard to claim 3: (Currently Amended)
FEEHRER does not explicitly disclose:
- wherein the crossbar comprises a plurality of input ports and a plurality of output ports, and the control information comprises information identifying which input ports of the crossbar are to be connected to which output ports of the crossbar to form the hardware pipeline.
However, BOESCH discloses:
- wherein the crossbar comprises a plurality of input ports and a plurality of output ports, and the control information comprises information identifying which input ports of the crossbar are to be connected to which output ports of the crossbar to form the hardware pipeline.
[0083]:
In another embodiment, a hardware accelerator embodiment supports efficient mapping of convolutional stages of deep neural network algorithms. The hardware accelerator engine includes a stream switch and a plurality of convolution accelerators. The stream switch has a first plurality of input bus ports and a second plurality of output bus ports. Each of the input and output bus ports has a plurality of distinct data communication conduits, and each of the input and output bus ports is arranged to pass data of a multipath bus. The stream switch is arranged to selectively couple each one of the first plurality of input bus ports to one or more of the second plurality of output bus ports such that data presented on individual lines of an input bus port is concurrently passed to corresponding individual lines of one or more output bus ports that are selectively coupled to the input bus port. Each one of the plurality of convolution accelerators has at least one input bus coupled to an output bus port of the stream switch. At least one output bus is coupled to an input bus port of the stream switch, or at least one input bus and at least one output bus are hard wired to respective output bus and input bus ports of the stream switch.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
One of ordinary skill would have motivation to combine FEEHRER and BOESCH that can use an on-chip an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic ( BOESCH [0123]).
In regard to claim 11: (Currently Amended)
FEEHRER does not explicitly disclose:
- wherein the neural network accelerator comprises a plurality of data input units configured to load the input data into the neural network accelerator,
- and the method further comprises selecting one of the plurality of data input units to load the input data into the neural network accelerator based on one or more characteristics of the input data and/or the hardware pipeline, and the control information comprises information identifying the selected data input unit.
However, BOESCH discloses:
- wherein the neural network accelerator comprises a plurality of data input units configured to load the input data into the neural network accelerator,
[0316]:
Process 200 proceeds to block 212, where the SoC configuration file and the SoC DCNN weights are uploaded to the SoC 110. The SoC 110 stores the SoC DCNN weights in its memory, such as SoC global memory 126, and when the SoC configuration file is loaded, the SoC 110 is enabled to perform image recognition on input image data.
- and the method further comprises selecting one of the plurality of data input units to load the input data into the neural network accelerator based on one or more characteristics of the input data and/or the hardware pipeline, and the control information comprises information identifying the selected data input unit.
[0306]:
he DCNN configurations 104 may also include configuration information deep learning frameworks that include fully formed neural network implementations such as Caffe, AlexNet, Theano, TensorFlow, GoogleLeNet, VGG19, ResNet, or other deep learning frameworks.
[0073]:
a CA configuration may be defined automatically using, for example, a holistic tool that starts from a DCNN description format such as Caffe' or TensorFlow [BRI: a convolution accelerator configuration described in a deep learning framework format such as TensorFlow can indeed represent a selection of one or more data input units to load input data into the neural network accelerator, based on one or more characteristics of the input data ]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
One of ordinary skill would have motivation to combine FEEHRER and BOESCH that can use an on-chip an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic ( BOESCH [0123]).
In regard to claim 12: (Currently Amended)
FEEHRER and BOESCH does not explicitly disclose:
- and the method further comprises selecting a processing order to be used by one or more of the at least one processing units for transmitting or receiving a tensor based on the hardware pipeline, and wherein the control information comprises information identifying the selected processing order.
However, Naga discloses:
- wherein at least one of the hardware processing units in the set is configurable to transmit or receive a tensor in a selected processing order of a plurality of selectable processing orders,
[0004]:
A neural network layer can have an associated set of kernels as well as an embedding layer for processing inputs to generate sets of vectors for training a neural network. Kernels can be represented as a tensor, i.e., a multi-dimensional array, of weights. As an example, embedding layers can process a set of inputs, such as inputs of image pixel data or activation values generated by a neural network layer. The set of inputs or set of activation values can also be represented as a tensor
[0048]:
Each processor of the multi-core processing unit 104 is configured to retrieve data elements stored in a memory of system 100. The memory can include multiple data shards 106a-106k that store data including elements 108a-108n. The data can include inputs, activations, gain values, or weight values corresponding to parameters or kernels of a matrix structure of weights.
- and the method further comprises selecting a processing order to be used by one or more of the at least one processing units for transmitting or receiving a tensor based on the hardware pipeline, and wherein the control information comprises information identifying the selected processing order.
[0056]:
the crossbar may be integrated in a processing pipeline of the chip's circuitry to enable each channel controller to read data from, and write data to, any channel of a high-bandwidth memory system.
[0081]:
the processing pipeline of a channel controller 202 is used to perform backward pass and forward pass operations with respect to an embedding layer of a neural network.
[0048]:
The data can include inputs, activations, gain values, or weight values corresponding to parameters or kernels of a matrix structure of weights.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, BOESCH and Naga.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches restrictions are not contravened when an order for the selected hardware units are determined.
One of ordinary skill would have motivation to combine FEEHRER, BOESCH and NAGA that can provide improve efficiency for neural network operations particularly load balancing( Naga[0006])
In regard to claim 16: (Currently Amended)
FEEHRER does not explicitly disclose:
- A method of configuring a neural network accelerator to implement a neural network, the neural network comprising a plurality of layers, each layer configured to receive input data and perform one or more neural network operations on the received input data, the method comprising:
- grouping the neural network operations of the neural network into one or more sequences of neural network operations, each sequence of neural network operations being executable by a combination of hardware processing elements;
- and executing the method as set forth in claim 1 for each sequence of neural network operations.
However, BOESCH discloses:
- A method of configuring a neural network accelerator to implement a neural network, the neural network comprising a plurality of layers, each layer configured to receive input data and perform one or more neural network operations on the received input data, the method comprising:
[Abstract]:
Embodiments are directed towards a hardware accelerator engine that supports efficient mapping of convolutional stages of deep neural network algorithms. The hardware accelerator engine includes a plurality of convolution accelerators, and each one of the plurality of convolution accelerators includes a kernel buffer, a feature line buffer, and a plurality of multiply-accumulate (MAC) units.
[0069]:
When state-of-the-art DCNNs are implemented on conventional, non-mobile hardware platforms, it is known that such DCNNs produce excellent results. Such DCNNs, however, require deeper topologies with many layers, millions of parameters, and varying kernel sizes.
[0073]:
a CA configuration may be defined automatically using, for example, a holistic tool that starts from a DCNN description format such as Caffe' or TensorFlow.
[0075]:
In a first embodiment, a hardware accelerator engine supports efficient mapping of convolutional stages of deep neural network algorithms.
[0074]:
The DSP's perform pooling (e.g., max pooling, average pooling, etc.), nonlinear activation, cross-channel response normalization, and classification representing a selected fraction of the total DCNN computation in an architecture that is flexible and amenable to future algorithmic evolutions
- grouping the neural network operations of the neural network into one or more sequences of neural network operations,
[0156]:
A different acceptably optimal configuration of CAs 600 in the CAF 400 is determined for each DCNN layer. These configurations may be determined or adjusted using a holistic tool that starts with a DCNN description format, such as Caffe′ or TensorFlow. [BRI: TensorFlow provides mechanisms to group multiple neural network operations into a single logical unit or sequence, which can be executed together and optimized for hardware execution]
[0151]:
The CAF400 utilizes unidirectional links to transport data streams via a configurable, fully connected switch to or from different kinds of source or sink components. For example, the configurable fully connected switch, which is described in more detail in conjunction with FIG. 5, can transport data via direct memory accesses (DMAs) to the SoC global memory 126, I/O interfaces (e.g., cameras), and various types of accelerators (e.g., convolution accelerator (CA)). In some cases, the CAF 400 is configured at boot time based on information received from a particular SoC configuration tool, and the CAF 400 is re-configured during run time based on defined DCNN layers and topology or information received from one or more DSPs 138, applications processor 128, or the like.
- each sequence of neural network operations being executable by a combination of hardware processing elements;
[0074]:
The DSP's perform pooling (e.g., max pooling, average pooling, etc.), nonlinear activation, cross-channel response normalization, and classification representing a selected fraction of the total DCNN computation in an architecture that is flexible and amenable to future algorithmic evolutions
[0123]:
The high-performance, energy efficient hardware accelerated DCNN processor described herein includes an energy efficient set of DCNN hardware convolution accelerators that support kernel decompression, fast data throughput, and efficient mathematical operation. The processor also includes an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic, and a power efficient array of DSPs that support complete, real-world computer vision applications.
[0152]:
The CAF 400 allows for the definition of a selectable number of concurrent, virtual processing chains at run time
[0084]:
one or more aspects of a design time parametric, run-time reconfigurable hardware accelerator interconnect framework that supports data-flow based processing chains [BRI: dataflow processing chains can indeed represent the sequence of neural network operations as executable by a combination of hardware processing elements]
- and executing the method as set forth in claim 1 for each sequence of neural network operations.
[0146]:
DSPs 138 can operate concurrently (e.g., in parallel) with the operations of CAs in the CAF 400 and concurrently (e.g., in parallel) with data transfers, which may be synchronized by way of interrupts, mailboxes, or some other synchronization mechanism for concurrent execution.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
One of ordinary skill would have motivation to combine FEEHRER and BOESCH that can use an on-chip an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic ( BOESCH [0123]).
In regard to claim 17: (Original)
FEEHRER discloses:
- A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform the method as set forth in claim 1.
[0065]:
the FAM SP CPU 154 in one embodiment executes instructions stored in an additional non-transitory memory connected to it to perform some or all of the following management functions:
In regard to claim 19: (Original)
FEEHRER discloses:
- A computing-based device comprising one or more processors configured to perform the method as set forth in claim 1.
[0014]:
The example non-limiting technologies herein permit the fabric attached memory to be of variable size, and provide address mapping and memory access request distribution techniques for ensuring that the fabric attached memory capacity is fully utilized. For example, an application running on a “source GPU” (i.e., a computing device that wishes to access the fabric attached memory)
In regard to claim 20: (New)
FEEHRER does not explicitly disclose:
- wherein forming the hardware pipeline from the hardware processing units comprises connecting inputs and outputs of the selected set of hardware processing units such that the output of one hardware processing unit of the selected set of hardware processing units becomes the input to another hardware processing unit of the selected set of hardware processing units
However, BOESCH discloses:
- wherein forming the hardware pipeline from the hardware processing units comprises connecting inputs and outputs of the selected set of hardware processing units such that the output of one hardware processing unit of the selected set of hardware processing units becomes the input to another hardware processing unit of the selected set of hardware processing units
[0083]:
In another embodiment, a hardware accelerator embodiment supports efficient mapping of convolutional stages of deep neural network algorithms. The hardware accelerator engine includes a stream switch and a plurality of convolution accelerators. The stream switch has a first plurality of input bus ports and a second plurality of output bus ports. Each of the input and output bus ports has a plurality of distinct data communication conduits, and each of the input and output bus ports is arranged to pass data of a multipath bus. The stream switch is arranged to selectively couple each one of the first plurality of input bus ports to one or more of the second plurality of output bus ports such that data presented on individual lines of an input bus port is concurrently passed to corresponding individual lines of one or more output bus ports that are selectively coupled to the input bus port. Each one of the plurality of convolution accelerators has at least one input bus coupled to an output bus port of the stream switch. At least one output bus is coupled to an input bus port of the stream switch, or at least one input bus and at least one output bus are hard wired to respective output bus and input bus ports of the stream switch.
[0152]:
The CAF 400 allows for the definition of a selectable number of concurrent, virtual processing chains at run time.
[BRI: the connectivity from input port to output port forms the connecting hardware units to form the pipeline]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
One of ordinary skill would have motivation to combine FEEHRER and BOESCH that can use an on-chip an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic ( BOESCH [0123]).
Claim 18 is rejected under 35 U.S.C. 103 unpatentable over
John FEEHRER et.al. (hereinafter FEEHRER) US 2021/0133122 A1,
In view of Thomas BOESCH et.al. (hereinafter BOESCH) US 2018/0189641 A1,
In regard to claim 18: (Currently Amended)
FEEHRER discloses:
- the neural network accelerator comprising a plurality of hardware processing units
and a crossbar coupled to each hardware processing unit of the plurality of hardware processing units
[0157]:
As shown in FIG. 18, the GPU 102 includes an Input/Output (I/O) unit 305, a front end unit 315, a scheduler unit 320, a work distribution unit 325, a hub 330, a crossbar (Xbar) 370, one or more general processing clusters (GPCs) 350, and one or more partition units 380. The GPU 102 may be connected to a host processor or other PPUs 300 via one or more high-speed NVLINK™ 310 interconnects forming an interconnect fabric
PNG
media_image1.png
672
517
media_image1.png
Greyscale
[0167]:
FIG. 19 illustrates a GPC 350 of the GPU 102 of FIG. 18, in accordance with an embodiment. As shown in FIG. 19, each GPC 350 includes a number of hardware units for processing tasks. In an embodiment, each GPC 350 includes a pipeline manager 410, a pre-raster operations unit (PROP) 415, a raster engine 425, a work distribution crossbar (WDX) 480, a memory management unit (MMU) 490, and one or more Data Processing Clusters (DPCs) 420.
- each hardware processing unit comprising hardware to accelerate performing one or more neural network operations on received data
[0156]:
As discussed above, one or more GPUs 102 as shown may be configured to accelerate thousands of High Performance Computing (HPC), data center, and machine learning applications. The GPU 102 may be configured to accelerate numerous deep learning systems and applications
FEEHRER does not explicitly disclose:
- the computing- based device comprising one or more processors configured to: to: obtain a sequence of one or more neural network operations to be performed by the neural network accelerator on the input data;
- select a set of hardware processing units from the plurality of hardware processing units to perform the one or more neural network operations
- determine an order of the selected set of hardware processing units to perform the one or more neural network operations in accordance with the sequence;
- and provide the neural network accelerator with control information that causes the crossbar of the neural network accelerator to form a hardware pipeline of the selected set of hardware processing units in the determined order to process the input data so as to configure the neural network accelerator to process the input data via the hardware pipeline of the selected set of hardware processing units in the determined order.
However, BOESCH discloses:
- the computing- based device comprising one or more processors configured to: obtain a sequence of one or more neural network operations to be performed by the neural network accelerator on the input data;
- A computer-based device for configuring a neural network accelerator to process input data,
[0157] :
FIG. 4 is an embodiment of configurable accelerator framework (CAF) 400, such as the image and deep convolutional neural network (DCNN) co-processor subsystem 400 of FIG. 3
[0083] :
In another embodiment, a hardware accelerator embodiment supports efficient mapping of convolutional stages of deep neural network algorithms [BRI: hardware accelerator engine with a configurable framework that supports deep neural networks (DNNs) can indeed be considered a computer-based device for configuring and operating a neural network accelerator to process input data]
[0067]:
In an exemplary architecture, two or more (e.g., eight) digital signal processor (DSP) clusters are formed in a system on chip (SoC). Each DSP cluster may include two or more DSP's,
[0070] :
In the exemplary architecture a design time configurable accelerator framework (CAF) includes unidirectional links transporting data streams via a configurable, fully connected switch to, from, or to and from source devices and sink devices. The source and sink devices may include any one or more of DMA's, input/output (I/O) interfaces (e.g., multimedia, satellite, radar, etc.), and various types of accelerators including one or more convolution accelerators (CA).
[0142]:
Each DSP cluster 122, 140 includes a plurality (e.g., two) of DSPs 142, 152, a plurality (e.g., two) of local DSP crossbar switches 144, 154, and a DSP cluster crossbar switch 145, 155. Each DSP 142, 152 in a particular cluster communicates with each other via the DSP cluster crossbar switch 145, 155.
[0139]:
the applications processor 128 may coordinate the reconfiguration of the CAF 400 or DSPs 138 based on the configuration file, which itself may be based on the DCNN layers and topology.
[0271] :
in some cases, when two or more kernels are processed in parallel and the lines are not interleaved, data from the CA adder tree 622 is passed through the CA output interface 608 as a fixed sequence. That is, in the fixed sequence, the result for a first kernel may be passed through the CA output interface 608, then the result for a second kernel, and so on. In this case, the output data can be handled in a manner similar to the way that batch size is used for kernel data and input feature data. For example, if four (4) kernels are processed in parallel, the output feature data will be processed as if a batch size equals four (4). Correspondingly in these cases, intermediate data passed through the first CA input interface 602 will also maintain a same sequence.
[0074]:
The DSP's perform pooling (e.g., max pooling, average pooling, etc.), nonlinear activation, cross-channel response normalization, and classification representing a selected fraction of the total DCNN computation in an architecture that is flexible and amenable to future algorithmic evolutions [ BRI: A sequence of NN operations on a convolution accelerator typically includes convolution and activation, and in many cases they are fused into a single hardware stage for performance gains ].
- select a set of hardware processing units from the plurality of hardware processing units to perform the one or more neural network operations
[0152]:
The CAF 400 allows for the definition of a selectable number of concurrent, virtual processing chains at run time. The CAF 400 also includes a full featured back pressure mechanism to control data flow to the various components of the framework. The CAF 400 is arranged for stream multicasting operations, which enable the reuse of a data stream at multiple block instance
[0151]:
the CAF 400 is re-configured during run time based on defined DCNN layers and topology or information received from one or more DSPs 138, applications processor 128, or the like. [BRI: A virtual processing chain is a logical pipeline that can be instantiated from a set of physical processing units. With reconfigurable CAF, plurality set of hardware units can be selected]
[0198]:
The selection mechanism 508 is directed according to stream switch configuration logic 510. The stream switch configuration logic 510 determines at run time which input port 504 shall supply data to the associated output port, and based on the determination, the stream switch configuration logic 510 forms an appropriate selection signal that is passed to the data switch 506. The stream switch configuration logic 510 operates at run time and in real time. The stream switch 510 may take direction from CAF control registers, from a DSP of the DSP cluster 122 (FIG. 3), from the application processor 128, or from some other control device. In addition, the stream switch configuration logic 510 may also take direction from message/command logic 512.
- determine an order of the selected set of hardware processing units to perform the one or more neural network operations in accordance with the sequence;
[0278]:
Using the feature data, kernel data, and intermediate data, the CA 600 performs one or more sequences of serial or parallel convolution operations to produce output data.
[0169]:
The DMA engines 406 are arranged to provide bidirectional channels for input data flow, output data flow, or input and output data flow. In these cases, substantial quantities of data is passed into the CAF 400, out from the CAF 400, or into and out from the CAF 400
[0157]:
The CAF 400 includes a stream switch 500 that provides a design time parametric, run-time reconfigurable accelerator interconnect framework to support data-flow based processing chains. [BRI: in both serial and parallel convolution chains, the data flow defines a processing order. In serial chains, this is a strict sequential order; in parallel chains, it’s a combined order after merging branches]
PNG
media_image2.png
541
576
media_image2.png
Greyscale
[0164]:
Continuing in the description of FIG. 4, in addition to the stream switch 500, the CAF 400 may also include a system bus interface module 404. The system bus interface module 404 provides an interface to other modules of SoC 110. As shown in the exemplary embodiment of FIG. 3, the CAF 400 is coupled to the secondary communication bus 166. In other cases, the CAF 400 may be coupled to the primary communication bus 132 or some other communication mechanism. Control information may be passed unidirectionally or bidirectionally through the system bus interface module 404 of the CAF 400. Such interface is used to provide a host processor (e.g., DSP of DSP cluster 130, applications processor 128, or another processor) access to all of the CAF control registers 402,
[0144]:
Additionally, the global DSP cluster crossbar switch 150 is communicatively coupled to a system bus 166 (e.g., secondary communications bus, xbar—SoC crossbar switch, or the like), which enables each DSP to communicate with other components of the SoC 110
[0161]:
The stream switch 500 is a unidirectional interconnection structure formed with a plurality of unidirectional “stream links.” The stream links are arranged to transport multibit data streams from accelerators, interfaces, and other logic modules to the stream switch 500 and from the stream switch 500 to accelerators, interfaces, and other logic modules
- and provide the neural network accelerator with control information that causes the crossbar of the neural network accelerator to form a hardware pipeline of the selected set of hardware processing units in the determined order to process the input data so as to configure the neural network accelerator to process the input data via the hardware pipeline of the selected set of hardware processing units in the determined order.
[0157]:
The CAF 400 includes a stream switch 500 that provides a design time parametric, run-time reconfigurable accelerator interconnect framework to support data-flow based processing chains. [BRI: A processing chain is a “hardware pipeline”][0244]:
At the start of a line in first cycle, a first CA MAC unit 620 of each of three clusters (i.e., three 1.sup.st MAC units) performs calculations of the first column for the first output value of the first line. [BRI: CA is a convolution accelerator. This does exploit pipeline and parallelism for efficiency. The “determined order” implies that the data flow is synchronized across clusters. For example, the first cluster’s accelerator might process the first line, then pass it to the second cluster’s first accelerator, and so on. This combines line-based streaming, parallel cluster processing, and pipelined dataflow to maximize throughput and minimize idle cycles]
[0158]:
When integrating known hardware data path accelerators, the inventors have recognized that system designers must often choose between tightly coupled or loosely coupled architectures. Based on this choice, particular programmable elements (e.g., processors, DSPs, programmable logic controllers (PLC's), and other programmable devices) deployed in such a device (e.g., a system on chip) would be required to adopt a suitable, but limiting programming model.
[0158]:
This is undesirable and limiting, however, because the hardwired data paths provide little if any flexibility at run-time,
[0159]:
To overcome these limitations, a reconfigurable data transfer fabric of the stream switch 500
[0144]:
the global DSP cluster crossbar switch 150 is communicatively coupled to a system bus 166 (e.g., secondary communications bus, xbar—SoC crossbar switch, or the like), which enables each DSP to communicate with other components of the SoC 110. For example, each DSP 142, 152 can communicate with one or more components (e.g., one or more convolution accelerators) of the CAF 400 or access an SoC global memory 126 via the system bus 166.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
One of ordinary skill would have motivation to combine FEEHRER and BOESCH that can use an on-chip an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic ( BOESCH [0123]).
Claims 4-8 are rejected under 35 U.S.C. 103 unpatentable over
John FEEHRER et.al. (hereinafter FEEHRER) US 2021/0133122 A1,
In view of Thomas BOESCH et.al. (hereinafter BOESCH) US 2018/0189641 A1,
In view of Rahul Nagarajan et.al. (hereinafter Naga) US 2021/0303978 A1.
further in view of Anuja Naik et.al. (hereinafter Naik) (NoC) using heterogeneous circuit switched routers, IEEE 2016 International Conference on VLSI Systems, Architectures, Technology and Applications (VLSI-SATA).
In regard to claim 4: (Original)
FEEHRER, BOESCH and Naga do not explicitly disclose:
- wherein the neural network accelerator comprises a register for each output port, and providing the control information to the neural network accelerator comprises causing a value to be written each register that identifies which input port of the plurality of input ports is to be connected to the corresponding output port.
However, Naik discloses:
- wherein the neural network accelerator comprises a register for each output port, and providing the control information to the neural network accelerator comprises causing a value to be written each register that identifies which input port of the plurality of input ports is to be connected to the corresponding output port.
In [1, Page 1]:
With increase in Very Large Scale Integration (VLSI) density, it is now possible to integrate general purpose processors, memory blocks, application specific intellectual property blocks (IP), digital signal processor (DSP), Graphic processor unit (GPU) and mixed signal functions on a single system-on-chip (SoC).
In [1, Page 1]:
With more applications that require battery powered embedded system units, the energy and area efficiency of the SoC is a very important factor.
In [IV, Page 3]:
Circuit switched router
We propose the concept of lane division multiplexing (LDM)
In [IV, Page 3]:
Using LDM, a single port is segmented into smaller sets of bus which can be used by different data streams simultaneously. Our implementation terms a router as R(5,4) consist of 5 ports where each port is divided into 4 lanes of equal size in one direction. For example, the router shown in Figure 4 has eight lanes per port with four incoming and four outgoing lanes. We recognize switching network inside a router consumes major silicon area. To reduce this silicon area and power dissipation, we propose using multistage CLOS network where a single CLOS switch is made up of multiple small Crossbar switches.
PNG
media_image3.png
397
431
media_image3.png
Greyscale
In [IV, Page 4;In Input allocation unit:
Input allocation unit checks the incoming four data arriving to a router port and allocate appropriate lane to each of them. The allocation depends upon the destined direction. (i.e. data destined to go in the South will be allocated to that particular lane). The unit stores incoming data in the temporary buffers till the routing decision takes place. The allocation algorithm checks for the destination address of each input and then send it to appropriate lane and set the flag “high” for that channel. A higher flag for the lane suggests that the particular lane reserved to transmit the message cannot be used by any other data until the previous transmission is completed. Flags for all the lanes cleared every time before initiating a data transfer. Once all four input channels of a single port are allocated, data can travel to the desired output ports simultaneously.
(BRI: the allocation unit temporary buffer has the destination address of the routing( output port)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, BOESCH, Naga and Naik.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches restrictions are not contravened when an order for the selected hardware units are determined.
Naik teaches routing destination address within a CLOS-based circuit switched router in which CLOS is formed as multi-stage crossbar.
One of ordinary skill would have motivation to combine FEEHRER, BOESCH, Naga and Naik that can provide improved energy efficiency for the CLOS router that uses multiple crossbar switches (Naik [Abstract, Page 1]).
In regard to claim 5: (Original)
FEEHRER, BOESCH and Naga do not explicitly disclose:
- wherein each input port is allocated a number and the value written to a register is the number of the input port to be connected to the corresponding output port.
However, Naik discloses:
- wherein each input port is allocated a number and the value written to a register is the number of the input port to be connected to the corresponding output port.
In [IV A, Page 3]:
To simplify our design, we allocated each small Crossbar of the first stage to different ports – North, South, East, West and Tile. Figure 5 shows an example where each Crossbar can carry simultaneously 4 inputs from one direction and send them to 4 different directions except the one coming from the tile. We assume that data will not backtrack to the same router from where it arrived from.
PNG
media_image4.png
216
448
media_image4.png
Greyscale
(BRI: A tile is a processing elements)
In [IV, Page 4;In Input allocation unit:
Input allocation unit checks the incoming four data arriving to a router port and allocate appropriate lane to each of them. The allocation depends upon the destined direction. (i.e. data destined to go in the South will be allocated to that particular lane). The unit stores incoming data in the temporary buffers till the routing decision takes place. The allocation algorithm checks for the destination address of each input and then send it to appropriate lane and set the flag “high” for that channel. A higher flag for the lane suggests that the particular lane reserved to transmit the message cannot be used by any other data until the previous transmission is completed. Flags for all the lanes cleared every time before initiating a data transfer. Once all four input channels of a single port are allocated, data can travel to the desired output ports simultaneously.
(BRI: the allocation unit temporary buffer has the destination address of each input port is the number of the input port in the routing)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, BOESCH, Naga and Naik.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches restrictions are not contravened when an order for the selected hardware units are determined.
Naik teaches routing destination address within a CLOS-based circuit switched router in which CLOS is formed as multi-stage crossbar.
One of ordinary skill would have motivation to combine FEEHRER, BOESCH, Naga and Naik that can provide improved energy efficiency for the CLOS router that uses multiple crossbar switches (Naik [Abstract, Page 1]).
In regard to claim 7: (Original)
FEEHRER, BOESCH and Naga do not explicitly disclose:
- prior to providing the neural network accelerator with the control information, determining whether the control information is valid, and only providing the neural network accelerator with the control information if it is determined that the control information is valid.
However, Naik discloses:
- prior to providing the neural network accelerator with the control information, determining whether the control information is valid, and only providing the neural network accelerator with the control information if it is determined that the control information is valid.
In [1, Page 1]:
In the circuit switching router, a path from source to destination is established before the transmission of data and it cannot be allocated to other resources till the desired data transmission is completed,
In [IV, Page 4;In Input allocation unit:
Input allocation unit checks the incoming four data arriving to a router port and allocate appropriate lane to each of them. The allocation depends upon the destined direction. (i.e. data destined to go in the South will be allocated to that particular lane). The unit stores incoming data in the temporary buffers till the routing decision takes place. The allocation algorithm checks for the destination address of each input and then send it to appropriate lane and set the flag “high” for that channel. A higher flag for the lane suggests that the particular lane reserved to transmit the message cannot be used by any other data until the previous transmission is completed. Flags for all the lanes cleared every time before initiating a data transfer. Once all four input channels of a single port are allocated, data can travel to the desired output ports simultaneously.
( BRI: A flag status of “0” means the transmission is “valid” that no blocking has occurred)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, BOESCH, Naga and Naik.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches restrictions are not contravened when an order for the selected hardware units are determined.
Naik teaches routing destination address within a CLOS-based circuit switched router in which CLOS is formed as multi-stage crossbar.
One of ordinary skill would have motivation to combine FEEHRER, BOESCH, Naga and Naik that can provide improved energy efficiency for the CLOS router that uses multiple crossbar switches (Naik [Abstract, Page 1]).
In regard to claim 8: (Original)
FEEHRER, BOESCH, and Naga do not explicitly disclose:
- wherein it is determined that the control information is valid only if, when the output of a first hardware processing unit is to be the input to a second hardware processing unit, the control information indicates that the input port of the crossbar coupled to the output of the first hardware processing unit is to be connected to the output port of the crossbar coupled to the input of the second hardware processing unit.
However, Naik discloses:
- wherein it is determined that the control information is valid only if, when the output of a first hardware processing unit is to be the input to a second hardware processing unit, the control information indicates that the input port of the crossbar coupled to the output of the first hardware processing unit is to be connected to the output port of the crossbar coupled to the input of the second hardware processing unit.
In [1, Page 1]:
In the circuit switching router, a path from source to destination is established before the transmission of data and it cannot be allocated to other resources till the desired data transmission is completed,
In [IV, Page 4]:In Input allocation unit:
Input allocation unit checks the incoming four data arriving to a router port and allocate appropriate lane to each of them. The allocation depends upon the destined direction. (i.e. data destined to go in the South will be allocated to that particular lane). The unit stores incoming data in the temporary buffers till the routing decision takes place. The allocation algorithm checks for the destination address of each input and then send it to appropriate lane and set the flag “high” for that channel. A higher flag for the lane suggests that the particular lane reserved to transmit the message cannot be used by any other data until the previous transmission is completed. Flags for all the lanes cleared every time before initiating a data transfer. Once all four input channels of a single port are allocated, data can travel to the desired output ports simultaneously.
( BRI: A flag status of “0” means the transmission is “valid” that no blocking has occurred)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, BOESCH, Naga and Naik.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches restrictions are not contravened when an order for the selected hardware units are determined.
Naik teaches routing destination address within a CLOS-based circuit switched router in which CLOS is formed as multi-stage crossbar.
One of ordinary skill would have motivation to combine FEEHRER, BOESCH, Naga and Naik that can provide improved energy efficiency for the CLOS router that uses multiple crossbar switches (Naik [Abstract, Page 1]).
Claims 2, 6, 10, and 13 are rejected under 35 U.S.C. 103 unpatentable over
John FEEHRER et.al. (hereinafter FEE) US 2021/0133122 A1
In view of Thomas BOESCH et.al. (hereinafter BOESCH) US 2018/0189641 A1.
In view of Rahul Nagarajan et.al. (hereinafter Naga) US 2021/0303978 A1.
further in view of Tirumale Ramesh et.al. (hereinafter Ramesh) US 8103853 B2.
In regard to claim 2: (Original)
FEEHRER and BOESCH do not explicitly disclose:
- wherein the control information comprises information identifying the selected set of hardware processing units and information identifying the determined order of the selected set of hardware processing units.
However, Ramesh discloses:
- wherein the control information comprises information identifying the selected set of hardware processing units and information identifying the determined order of the selected set of hardware processing units.
In [Col 5, lines 9-14]:
FIG. 5 shows a block diagram identifying different types of fabric element cells which may be instantiated on a chip. The base entity of a fabric element may comprise a "fabric element cell" termed as FEC. The fluidity in the fabric may be demonstrated by the flexible residency of the fabric element cell within the physical entity.
In [Col 6, line 46-49]:
the fabric morphing control 12 may provide global control of fabric element cells morphing from global application instances to a processor micro-architecture 29( shown in FIG. 6).
In [Col 3, line 29-35]:
The soft processor 14 may comprise single core or multiple cores 15. With multiple cores, cores may be allocated and reallocated at run-time to optimize for performance based on the load balancing on these core workloads. Any custom cores for specific functions may be combined into a group of a single entity for aggregation of processing powers from the cores.
(BRI: the fabric morphing control offers global control over how fabric element cells group and reconfigure and provides the selection of hardware units to match the data flow or control flow of the running hardware)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEHRER, BOESCH, and Ramesh.
FEE teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Ramesh teaches reconfigurable fabric with fabric switch forming a crossbar and facilitate forming a pipelining of execution units in the determined order and in fact demonstrates a highly powerful overarching accelerator architecture (See [Col 7, lines 61-67]).
One of ordinary skill would have motivation to combine FEEHRER, BOESCH, Naga and Ramesh and with a cognitive brain (cognitive processor) added within a smart switch, the cognitive control for the morphing process may add greater optimization of resource utilization and performance (Ramesh [Col 5, lines 48-51]).
In regard to claim 6: (Original)
FEEHRER, BOESCH and Naga do not explicitly disclose:
- wherein when a hardware processing unit of the plurality of hardware processing units does not from part of the set of hardware processing units then a predetermined value is written to the register corresponding to the output port connected to that hardware processing unit to indicate that the hardware processing unit is to be disabled.
However, Ramesh discloses:
- wherein when a hardware processing unit of the plurality of hardware processing units does not from part of the set of hardware processing units then a predetermined value is written to the register corresponding to the output port connected to that hardware processing unit to indicate that the hardware processing unit is to be disabled.
In [Col 3, lines 54-67]:
FIG. 2 comprises an architecture diagram of an distributed virtual connectivity switch VS including cognitive processors 93, switches 9, edge caches 10, distributed reconfigurable processors 11, network interface 4a, global cache control 16, and storage 18a. The integrated virtual connective switch VS may comprise a mesh connected multi-processing architecture with distributed processor switch elements 9 having at least four ports per element. Each distributed element may comprise a cognitive processor 93 and a switch element 9. The cognitive processors 93 and 4-port switches 9 may be orthogonally laid and distributed. Every cognitive processor 93 may take intermediate decisions and pass it onto next cognitive processor 93 via switch 9. It may be possible to bypass a series of switches 9 for one cognitive processor 93
In [Col 4, line 1]:
to virtually connect to another.
In [Col 4, lines 4-8]:
At each switch interconnection, an edge cache 10 may be inserted that caches intermediate decisions and data by the cognitive processor 93 to effectively be used by any other cognitive processor 93 without having to access data from the source.
(BRI: the edge cache and matrix switch combination can act as a hardware-implemented logic gate/multiplexer that ensures system stability by providing a default value when an expected processing unit is absent or offline. The cache itself might store this predetermined value or act as the buffer/pathway for its insertion into the register)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEHRER, BOESCH, and Ramesh.
FEE teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Ramesh teaches reconfigurable fabric with fabric switch forming a crossbar and facilitate forming a pipelining of execution units in the determined order and in fact demonstrates a highly powerful overarching accelerator architecture (See [Col 7, lines 61-67]).
One of ordinary skill would have motivation to combine FEEHRER, BOESCH, Naga and Ramesh and with a cognitive brain (cognitive processor) added within a smart switch, the cognitive control for the morphing process may add greater optimization of resource utilization and performance (Ramesh [Col 5, lines 48-51]).
In regard to claim 10: (Original)
FEEHRER, BOESCH and Naga do not explicitly disclose:
- wherein the set of hardware processing units are selected such that each hardware processing unit of the set of hardware processing units is only used once in performing the sequence of one or more neural network operations.
However, Ramesh discloses:
- wherein the set of hardware processing units are selected such that each hardware processing unit of the set of hardware processing units is only used once in performing the sequence of one or more neural network operations.
In [Col 5, lines 9-14]:
FIG. 5 shows a block diagram identifying different types of fabric element cells which may be instantiated on a chip. The base entity of a fabric element may comprise a "fabric element cell" termed as FEC. The fluidity in the fabric may be demonstrated by the flexible residency of the fabric element cell within the physical entity.
In [0029]:
certain ML models may be designated to execute on certain cores of the target hardware. For example, a ML model which uses more processing resources may be assigned to execute on a certain ML core which may have an increased amount of processing power,
(BRI: the designation of certain Machine Learning (ML) models or parts of models to specific hardware cores can mean that a given hardware processing unit is used exclusively for that specific operation or set of operations within a sequence that is not repetitive in nature)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEHRER, BOESCH, Naga and Ramesh.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches
Ramesh teaches reconfigurable fabric with fabric switch forming a crossbar and facilitate forming a pipelining of execution units in the determined order and in fact demonstrates a highly powerful overarching accelerator architecture (See [Col 7, lines 61-67]).
One of ordinary skill would have motivation to combine FEEHRER, BOESCH, Naga and Ramesh and with a cognitive brain (cognitive processor) added within a smart switch, the cognitive control for the morphing process may add greater optimization of resource utilization and performance (Ramesh [Col 5, lines 48-51]).
In regard to claim 13: (Original)
FEEHRER, BOESCH and Naga do not explicitly disclose:
- wherein the control information further comprises information identifying a function and/or one or more operations to be implemented by one or more of the hardware processing units in the set of hardware processing units.
However, Ramesh discloses:
- wherein the control information further comprises information identifying a function and/or one or more operations to be implemented by one or more of the hardware processing units in the set of hardware processing units.
In [Col 6, line 46-49]:
the fabric morphing control 12 may provide global control of fabric element cells morphing from global application instances to a processor micro-architecture 29( shown in FIG. 6).
In [Col 3, line 29-35]:
The soft processor 14 may comprise single core or multiple cores 15. With multiple cores, cores may be allocated and reallocated at run-time to optimize for performance based on the load balancing on these core workloads. Any custom cores for specific functions may be combined into a group of a single entity for aggregation of processing powers from the cores.
(BRI: the fabric morphing control offers global control over how fabric element cells group and reconfigure)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, BOESCH, Naga and Ramesh.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
Naga teaches a configuration for connectivity and to account for restrictions to ensure validity of the connectivity.
Ramesh teaches reconfigurable fabric with fabric switch forming a crossbar and facilitate forming a pipelining of execution units in the determined order and in fact demonstrates a highly powerful overarching accelerator architecture (See [Col 7, lines 61-67]).
One of ordinary skill would have motivation to combine BOESCH, and Ramesh and with a cognitive brain (cognitive processor) added within a smart switch, the cognitive control for the morphing process may add greater optimization of resource utilization and performance (Ramesh [Col 5, lines 48-51]).
In regard to claim 14: (Currently Amended)
FEEHRER discloses:
- and providing the neural network accelerator with second control information that causes the crossbar of the neural network accelerator to form a second hardware pipeline of the selected second set of hardware processing units in the determined second order to process the second input data;
[0174] :
As shown in FIG. 20, the graphics processing pipeline 600 comprises a pipeline architecture that includes a number of stages. [BRI: Within the context of a pipeline architecture that includes number of stage, a plurality of hardware pipeline are represented].
[0186]:
in an embodiment, the device driver is configured to implement the graphics processing pipeline 600 utilizing the hardware of the GPU 102. [BRI: Within the context of a GPU 102, a second set of hardware processing units are represented].
[0086]:
Referring again to FIG. 6, as discussed above the amount of buffer space for shared memory 506 that is used by each channel controller 202 can be very different and can lead to underutilization of the scratchpad memory buffers 508. The underutilization results in lower batch sizes that can be processed for a given workload, leading to degraded or lower performance at system 100. To resolve the memory imbalance and improve the efficiency and utilization of the shared buffers, the system 100 is configured to allocate space in the circular buffers 510 based at least on a latency of the memory accesses observed in an example processing pipeline of each channel controller.
[0089]:
The circular buffer unit 610 can determine an allocation of shared resources in the shared memory 506 by determining an amount of scratchpad buffer space to be used by the selected channel controller 202 and a VPU 604 of a processor 602 that performs a portion of the neural network computations. For example, the allocation of shared resources is determined based on latency of memory accesses observed in an example processing pipeline of each channel controller 202. Based on the determined allocation, a set of gradient vectors may be copied into an allocated space of buffer/bank 704 and operated on using the VPU 604, or the address handler unit 606 described above. In some implementations, the shared buffer space may be a recently deallocated entry in a buffer/bank 704 of shared memory 506. [BRI: Within the context of a GPU 102 with a plurality of processing pipeline (configured via crossbar switch) , a second order of processing is represented].
FEEHRER does not explicitly disclose:
- obtaining a second sequence of one or more neural network operations to be performed by the neural network accelerator on second input data;
- selecting a second set of hardware processing units from the plurality of hardware processing units to perform the one or more neural network operations of the second sequence;
- determining a second order of the second selected set of hardware processing units to perform the one or more neural network operations in accordance with the second sequence;
- wherein the second set of hardware processing units is the same as the set of hardware processing units, and the second determined order is different than the determined order.
However, BOESCH discloses:
- obtaining a second sequence of one or more neural network operations to be performed by the neural network accelerator on second input data;
[0187]:
In the embodiment, one or more of the input stream link ports may be desirably arranged to concurrently forward received data streams to one or multiple (multicast) output ports on the same clock cycle. [BRI: received data streams provide second data ]
[0157]:
run-time reconfigurable accelerator interconnect framework to support data-flow based processing chains. [ BRI:in the context of massively parallel processing systems, “plural processing chains” refers to multiple independent processing chains that operate in parallel. Each chain is a sequence of interconnected processing elements that perform computations on data, and the “plural” part means there are more than one such chain in the system]
- selecting a second set of hardware processing units from the plurality of hardware processing units to perform the one or more neural network operations of the second sequence;
[0152]:
The CAF 400 allows for the definition of a selectable number of concurrent, virtual processing chains at run time. The CAF 400 also includes a full featured back pressure mechanism to control data flow to the various components of the framework. The CAF 400 is arranged for stream multicasting operations, which enable the reuse of a data stream at multiple block instance
[0151]:
the CAF 400 is re-configured during run time based on defined DCNN layers and topology or information received from one or more DSPs 138, applications processor 128, or the like. [BRI: A virtual processing chain is a logical pipeline that can be instantiated from a set of physical processing units. With reconfigurable CAF, plurality set of hardware units can be selected]
[0198]:
The selection mechanism 508 is directed according to stream switch configuration logic 510. The stream switch configuration logic 510 determines at run time which input port 504 shall supply data to the associated output port, and based on the determination, the stream switch configuration logic 510 forms an appropriate selection signal that is passed to the data switch 506. The stream switch configuration logic 510 operates at run time and in real time. The stream switch 510 may take direction from CAF control registers, from a DSP of the DSP cluster 122 (FIG. 3), from the application processor 128, or from some other control device. In addition, the stream switch configuration logic 510 may also take direction from message/command logic 512.
- determining a second order of the second selected set of hardware processing units to perform the one or more neural network operations in accordance with the second sequence;
[0269]:
To address the undesirable idle states, the highly configurable CA 620 may optionally include an additional line interleaving mode. The additional line interleaving mode may be configured to apply multiple kernels to the same feature in a line-interleaved fashion. This line interleaving of feature data enables the CA 620 to process four 11×11 kernels in parallel on the same feature data. [ BRI: A well-known mechanism in the art. Line" refers to the data path or channel in the crossbar ach line can carry a different data stream or micro-batch and Interleaved means that instead of sending all data from one stage to the next in a strict sequential order, the system alternates or mixes the data from different stages or devices. In line-interleaved mode, the pipeline schedules work so that different devices or stages process different data lines in an alternating fashion. This avoids long idle periods and keeps the crossbar busy. Any interleaved path represents a second order used for execution]
- wherein the second set of hardware processing units is the same as the set of hardware processing units, and the second determined order is different than the determined order.
[0269]:
To address the undesirable idle states, the highly configurable CA 620 may optionally include an additional line interleaving mode. The additional line interleaving mode may be configured to apply multiple kernels to the same feature in a line-interleaved fashion. This line interleaving of feature data enables the CA 620 to process four 11×11 kernels in parallel on the same feature data. [ BRI: the configuration to form inter-leaved path itself can represent that these different paths may not be same (not using same set of hardware processing units within the processing chain]
[0280]:
The simple processing chain illustrated in FIG. 6E may be effective in some embodiments, but in other embodiments, resource allocation can be improved
[0161]:
The control information may be used to signal events, to reconfigure the processing chain itself
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account connectivity for restrictions to ensure validity of the connectivity.
In regard to claim 15: (Original)
FEEHRER does not explicitly disclose:
wherein the plurality of hardware processing units comprises one or more of:
- a convolution processing unit configured to accelerate convolution operations between input data and weight data,
- an activation processing unit configured to accelerate applying an activation function to data,
- an element-wise operations processing unit configured to accelerate performing one or more element-wise operations on a set of data,
- a pooling processing unit configured to accelerate applying a pooling function on data,
- a normalisation processing unit configured to accelerate applying a normalisation function to data,
- and an interleave processing unit configured to accelerate rearrangement of data
However, BOESCH discloses:
- a convolution processing unit configured to accelerate convolution operations between input data and weight data,
[0216]:
feature data is applied to the plurality of CA MAC units 620. Feature and kernel buffer data applied to the CA MAC units 620 is mathematically combined according to the convolutional operations [BRI: in the context of hardware accelerators, a convolution unit is indeed represents a combination of feature (activation) data, kernel (weight) buffer, and a parallel array of MAC units]
[0216]:
feature data is applied to the plurality of CA MAC units 620. Feature and kernel buffer data applied to the CA MAC units 620 is mathematically combined according to the convolutional operations [BRI: in the context of hardware accelerators, a convolution unit is indeed represents a combination of feature (activation) data, kernel (weight) buffer, and a parallel array of MAC units]
- an activation processing unit configured to accelerate applying an activation function to data,
[0227]:
To process a kernel of a convolution layer, each value (i.e., each pixel) of the input feature at a first position (e.g., upper right corner, upper left corner, or some other position) is multiplied with each corresponding value of the kernel, and the products are summed to generate one output result. The output result is immediately or later passed to an input of one or more subsequent tasks such as max pooling and non-linear activation [BRI: in the context of hardware accelerators, a activation unit is indeed represents a combination of feature (activation) data, kernel (weight) buffer, and a parallel array of MAC units, and activation]
- an element-wise operations processing unit configured to accelerate performing one or more element- wise operations on a set of data,
[0157]:
the CAF 400 includes a stream switch 500 that provides a design time parametric, run-time reconfigurable accelerator interconnect framework to support data-flow based processing chains.
[0281]:
Due to the flexibility provided by the configurable accelerator framework (CAF) 400 (FIG. 4), an in particular stream switch 500 (FIG. 5), throughput and resource efficiency may be improved by chaining and grouping multiple CA 600 units.
[0309]:
DMA descriptor chains generation; and acceptably optimal mapping and scheduling of DCNN execution on configurable accelerator framework and DSP clusters
[0306]:
The DCNN configurations 104 may also include configuration information deep learning frameworks that include fully formed neural network implementations such as Caffe, AlexNet, Theano, TensorFlow, GoogleLeNet, VGG19, ResNet, or other deep learning frameworks. [BRI: a tensorflow provides an element-wise operations]
- a pooling processing unit configured to accelerate applying a pooling function on data,
0306]:
The DCNN configurations 104 may also include configuration information deep learning frameworks that include fully formed neural network implementations such as Caffe, AlexNet, Theano, TensorFlow, GoogleLeNet, VGG19, ResNet, or other deep learning frameworks. [BRI: a tensorflow provides a pooling operation]
- a normalisation processing unit configured to accelerate applying a normalisation function to data,
[0157]:
the CAF 400 includes a stream switch 500 that provides a design time parametric, run-time reconfigurable accelerator interconnect framework to support data-flow based processing chains.
[0281]:
Due to the flexibility provided by the configurable accelerator framework (CAF) 400 (FIG. 4), an in particular stream switch 500 (FIG. 5), throughput and resource efficiency may be improved by chaining and grouping multiple CA 600 units.
[0309]:
DMA descriptor chains generation; and acceptably optimal mapping and scheduling of DCNN execution on configurable accelerator framework and DSP clusters
[0306]:
The DCNN configurations 104 may also include configuration information deep learning frameworks that include fully formed neural network implementations such as Caffe, AlexNet, Theano, TensorFlow, GoogleLeNet, VGG19, ResNet, or other deep learning frameworks. [BRI: a TensorFlow provides a normalization operation]
- and an interleave processing unit configured to accelerate rearrangement of data.
[0306]:
The DCNN configurations 104 may also include configuration information deep learning frameworks that include fully formed neural network implementations such as Caffe, AlexNet, Theano, TensorFlow, GoogleLeNet, VGG19, ResNet, or other deep learning frameworks. [BRI: a TensorFlow provides a interleaved operation
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine FEEHRER, and BOESCH.
FEEHRER teaches a hardware accelerator with set of hardware units for neural network acceleration and teaches forming a crossbar and facilitate forming a pipelining of execution units in the determined order.
BOESCH teaches a convolution accelerator and accelerator framework, a reconfigurable dataflow fabric and to account for restrictions to ensure validity of the connectivity.
One of ordinary skill would have motivation to combine FEEHRER and BOESCH that can use an on-chip an on-chip reconfigurable data transfer fabric that improves data reuse and reduces on-chip and off-chip memory traffic ( BOESCH [0123]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the
examiner should be directed to TIRUMALE KRISHNASWAMY RAMESH whose telephone number is (571)272-4605. The examiner can normally be reached by phone.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached on phone (571-272-3768). The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format.
For additional questions, contact the Electronic
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO
Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIRUMALE K RAMESH/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights