Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Objections
The disclosure is objected to because of the following informalities:
Paragraph [0035] refers to "The convolution units 106, the activation units 108, and the pooling units 110." Reference numeral 106 is established as "feature data," 108 as the "stream switch," and 110 as the "internal storage" (see paragraph [0029]-[0030]). These numerals cannot simultaneously designate different components. The correct numeral for these units may be 112 (hardware accelerators), or each unit may require its own distinct numeral.
Paragraph [0044] refers to "the stream switch 104." Reference numeral 104 designates the "external memory" (see paragraph [0029]). The stream switch is designated 108.
Paragraph [0044] refers to "the stream switch 105." Reference numeral 105 designates the "stream engine" (see paragraph [0044], first sentence). The stream switch is designated 108.
Paragraph [0046] refers to "one convolution unit 104." Reference numeral 104 designates the "external memory" (see paragraph [0029]), not a convolution unit. Additionally, the sentence "The new feature data is provided from one convolution unit 104 a next hardware accelerator 112" is missing the word "to" between "104" and "a next hardware accelerator."
Paragraph [0050] refers to "the external storage 110." Reference numeral 110 consistently designates the "internal storage" throughout the specification (see, e.g., paragraph [0029]-[0030]), and the sentence itself states that element 110 is implemented "within the neural network 102." The term "external" should read "internal."
Paragraph [0050] refers to "the stream switch 105." Reference numeral 105 designates the "stream engine," not the "stream switch." The stream switch is designated 108. This is the same error identified at paragraph [0044].
Paragraph [0056] refers to "the pooling unit 110." Reference numeral 110 designates the "internal storage" (see paragraph [0029]-[0030]). The pooling unit is designated 122 earlier in the same paragraph.
Paragraph [0056] refers to "method 1100." No method 1100 exists in the specification. The paragraph describes FIG. 17 and should reference "method 1700."
Paragraph [0056] describes both writing and reading operations under the same step number "204." These appear to be distinct operations and assigning the same step number to both is potentially confusing.
Paragraph [0062] refers to "stripes 134." Reference numeral 134 designates "batches" (see paragraph [0062]-[0064]). Stripes are designated 135 at the first use in the same paragraph.
Paragraph [0073] refers to "feature data 146." Reference numeral 146 designates the "write transformer unit" (see paragraph [0068]). Feature data is designated 106.
Paragraph [0085] refers to "the internal storage 120." Reference numeral 120 designates the "convolution accelerator" (see paragraph [0056]). The internal storage is designated 110.
Paragraph [0096] refers to "the control unit 150." Reference numeral 150 designates the "output buffers" (see paragraph [0069]). The control unit is designated 152.
Paragraph [0096] references "FIGS. 13A–13D." The specification does not include FIGS. 13A–13D; there is only a single FIG. 13. The correct cross-reference appears to be FIGS. 12A–12D, which depict the memory and registers of the internal storage.
Paragraph [0113] refers to "registers 160a-106g." Reference numeral "106g" does not exist. The registers are consistently in the 160 series and the reference should read "160a–160g."
Paragraph [0118] refers to "register 16 CE." This is not a valid reference numeral. Based on context, this should read "register 160e.".
Applicant should review the specification in details. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21-22, 38, and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over Munteanu et al.(US 20200126178 A1) in view of Liu et al. (US 20210390367 A1).
As to claim 21, Munteanu teaches a method, comprising:
receiving, at a neural network, feature data from a memory external to the neural network;
(See Munteanu paragraph [0035] "The CNN Engine 30 connects to a system bus 42 and can access main (DRAM) memory 40 into which images acquired by the system are written.")
Examiner Note: The CNN Engine 30 corresponds to the neural network, and DRAM memory 40 corresponds to the memory external to the neural network.
passing the feature data to an internal storage of the neural network; (See Munteanu paragraph [0044] "the image cache 31 is initially loaded with an input image/map from DRAM 40. Then all processing can be performed only using this image cache 31 with no need to access the external DRAM for image information until classification is complete.")
Examiner Note: The image cache 31 corresponds to the internal storage, receiving data from external DRAM 40.
storing the feature data in a memory of the internal storage; (See Munteanu paragraph [0062] "the image cache 31 comprises an interleaved memory architecture allowing reading and writing of blocks of pixels in one clock cycle. In the embodiment, there are 4 SRAM instances—SRAM0 to SRAM3.")
Munteanu does not explicitly teach "passing, with a read transformer unit of the internal storage, the feature data to a plurality of registers of the internal storage; and", "passing the feature data from the registers to a hardware accelerator of the neural network"
However, Liu teaches:
passing, with a read transformer unit of the internal storage, the feature data to a plurality of registers of the internal storage; and (See Liu paragraph [0103] " Input data selector 220 receives activation data from memory 176, and, based on a control signal from controller 210, sends the activation data to registers 230-X.sub.1, 230-X.sub.2, 230-X.sub.3, 230-X.sub.4, or registers 240-Y.sub.1, 240-Y.sub.2, 240-Y.sub.3, 240-Y.sub.4.")
Examiner Note: The MEU 200 including input data selector 220 and register sets corresponds to a read transformer unit that passes feature data from local memory 176 (memory of the internal storage) to a plurality of registers.
passing the feature data from the registers to a hardware accelerator of the neural network. (See Liu paragraph [0106] "Output data selector 250 receives expanded activation data from register set 230 and register set 240, and sends the expanded activation data to a plurality of PEs 180 based on a control signal from controller 210.")
Examiner Note: PEs 180 (processing elements including MAC units) correspond to the hardware accelerator.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the CNN Engine of Munteanu, which loads feature data from external DRAM into an internal image cache comprising SRAM instances and provides data to a convolution engine, to further incorporate the matrix expansion unit (MEU) of Liu, including its register sets and output data selector interposed between the local memory and the processing elements, because Liu expressly teaches that such a unit "advantageously reduces the memory footprint to that of the native convolution operation, reduces the memory bandwidth required for data movement, which increases the power efficiency at the system level, and takes advantage of the compute regularity of matrix multiplication, which can be more readily optimized in hardware" (Liu paragraph [0021]). A person of ordinary skill would have recognized that Munteanu's image cache already reduces external DRAM accesses by caching data locally (Munteanu paragraph [0044] "all processing can be performed only using this image cache 31 with no need to access the external DRAM for image information until classification is complete"), and that further incorporating Liu's register-based data reorganization between the local memory and the compute units would complement this caching by reformatting the cached data into the exact sequences needed by the processing elements, thereby further reducing redundant memory reads and improving overall throughput with a predictable and reasonable expectation of success.
As to claim 22, Munteanu as modified by Liu teaches the method of claim 21,
wherein passing the feature data from the registers to the hardware accelerator includes passing the feature data from the registers to one or more buffers of the internal storage. (See Munteanu paragraph [0068] "Including the output register (dout), there are three pipeline stages in this implementation.")
Examiner Note: The output register (dout) of the image cache functions as a buffer stage between the internal storage's data output and the convolution engine, corresponding to one or more buffers of the internal storage.
As to claim 38, Munteanu teaches a device comprising a neural network, the neural network including:
a stream engine configured to receive feature data from a memory external to the neural network; (see Munteanu paragraph [0036]"the configurable succession of feature extraction and classification operations to be performed by the CNN Engine 30 is determined by the CPU 50 by setting registers within the controller 60 via the system bus 42.", and see Munteanu paragraph [0039] "Data is read through the data input port (din) either from DRAM 40 via read controller 36")
Examiner Note: Read controller 36 receives data from external DRAM 40 and provides it to the CNN Engine 30, functioning as a stream engine.
a hardware accelerator; and (See Munteanu paragraph [0057] "The input data for each output map pixel and the weights are presented to the convolution engine 32.")
an internal storage configured to receive the feature data from the stream engine, (See Munteanu paragraph [0044] "the image cache 31 is initially loaded with an input image/map from DRAM 40. Then all processing can be performed only using this image cache 31 with no need to access the external DRAM for image information until classification is complete.")
the internal storage including: a memory configured store the feature data; (See Munteanu paragraph [0062] "the image cache 31 comprises an interleaved memory architecture allowing reading and writing of blocks of pixels in one clock cycle. In the embodiment, there are 4 SRAM instances—SRAM0 to SRAM3.")
Munteanu does not explicitly teaches "a plurality of registers coupled to the memory; and", and "a read transformer unit configured to read the feature data to the hardware accelerator including passing the feature data from the memory to the plurality of registers."
However, Liu teaches:
a plurality of registers coupled to the memory; and (See Liu paragraph [0104] "register set 230 includes four data selectors 230-M.sub.1, 230-M.sub.2, 230-M.sub.3 and 230-M.sub.4, and four registers 230-X.sub.1, 230-X.sub.2, 230-X.sub.3 and 230-X.sub.4, arranged in a shift-loop.")
a read transformer unit configured to read the feature data to the hardware accelerator including passing the feature data from the memory to the plurality of registers. (See Liu paragraph [0078] "MEU 200 is coupled to memory 176 and PEs 180, and converts or expands the original version of the IFM data (activations, “A”) to an IM2COL version, and provides the expanded IFM data (expanded activation sequences, “A.sub.i”) to PEs 180", and see Liu paragraph [0103] " Input data selector 220 receives activation data from memory 176, and, based on a control signal from controller 210, sends the activation data to registers 230-X.sub.1, 230-X.sub.2, 230-X.sub.3, 230-X.sub.4, or registers 240-Y.sub.1, 240-Y.sub.2, 240-Y.sub.3, 240-Y.sub.4.")
Examiner Note: The MEU 200 functions as a read transformer unit that reads data from memory 176 and passes it through register sets 230/240 before outputting to PEs 180 (hardware accelerator).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the CNN device of Munteanu, which includes a read controller functioning as a stream engine to receive feature data from external DRAM, an image cache as internal storage comprising SRAM instances, and a convolution engine as a hardware accelerator, to further incorporate the MEU structure of Liu including a plurality of registers coupled to local memory and a read transformer unit that passes data from memory through registers to processing elements, because Liu teaches that the MEU "is disposed inline between the memory and the CPU, specialized processor, hardware accelerator processing engine, etc., and converts the original version of the IFM matrix to an IM2COL version" (Liu paragraph [0021]), and that this inline data expansion using register sets arranged in shift loops with an output data selector enables the hardware accelerator to receive properly formatted expanded activation sequences without requiring the expanded data to be stored back into main memory (Liu paragraph [0078]: "MEU 200 is coupled to memory 176 and PEs 180, and converts or expands the original version of the IFM data (activations, 'A') to an IM2COL version, and provides the expanded IFM data (expanded activation sequences, 'Ai') to PEs 180"). A person of ordinary skill would have been motivated to add such a register-based read transformation stage to Munteanu's existing image cache architecture to achieve the same bandwidth and power efficiency benefits Liu identifies, with a reasonable expectation of success given that both references operate in the same field of CNN hardware acceleration with local SRAM-based storage feeding convolution compute units.
As to claim 40, Munteanu as modified by Liu teaches the device of claim 38,
wherein the internal storage includes a control unit configured to control the read transformer unit. (See Liu paragraph [0076] " HA 170 includes one or more controllers 172, communication bus interface 174, local memory 176 (e.g., SRAM, DRAM, etc.), one or more PEs 180, and one or more matrix expansion units (MEUs) 200. Controller 172 is coupled to communication bus interface 174, local memory 176, PEs 180 and MEU 200, and generally controls the components, functions, data flow, etc., of HA 170. In certain embodiments, controller 172 is an Arm Cortex-M33 microcontroller (MCU)", and see Liu paragraph [0102] "Controller 210 generally controls the components, data flow, etc., of MEU 200 in response to control signals received from controller 172.")
Claim(s) 23-29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Munteanu et al.(US 20200126178 A1) in view of Liu et al. (US 20210390367 A1) and Feinberg et al. (US 20190286975 A1).
As to claim 23, Munteanu as modified by Liu teaches the method of claim 21,
Munteanu-Liu does not explicitly teach "wherein passing the feature data to the plurality of registers includes passing, on each of a plurality of clock cycles, N data values to a first register of the plurality of registers"
However, Feinberg teaches:
wherein passing the feature data to the plurality of registers includes passing, on each of a plurality of clock cycles, N data values to a first register of the plurality of registers. (See Feinberg paragraph [0062] "the first row of data storage elements may be called a "header"", and See Feinberg paragraph [0071] "FIGS. 10B-10D depict a row-by-row loading of data values from the first input channel 702a into the 2-D shift register and a row-to-row shifting of the data values through the 2-D shift register.")
Examiner Note: On each clock cycle, a row of N data values (e.g., xn,1 through xn,4 in the embodiment) is loaded into the first row (header) of the 2-D shift register, which corresponds to passing N data values to a first register on each clock cycle.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined system of Munteanu and Liu to further incorporate the row-by-row data loading scheme of Feinberg, in which on each clock cycle a row of N data values is loaded into the first row (header) of a 2-D shift register, because Feinberg teaches that this serial row loading into a shift register structure enables efficient pipelining of convolution operations, explaining that "the first row of data storage elements may be called a 'header'" (Feinberg paragraph [0062]) and illustrating that "FIGS. 10B-10D depict a row-by-row loading of data values from the first input channel 702a into the 2-D shift register and a row-to-row shifting of the data values through the 2-D shift register" (Feinberg paragraph [0071]), with the result that "new data values may be stored in the nine data storage elements at clock cycle n+1 while the partial sum is stored. Therefore the computation of a new partial sum may be performed during every clock cycle" (Feinberg paragraph [0125]). A person of ordinary skill would have been motivated to adopt Feinberg's per-cycle N-value register loading approach within the register pipeline of Liu's MEU as combined with Munteanu's cache, because doing so would enable continuous pipelined data delivery to the convolution hardware on every clock cycle, thereby maximizing computational throughput, and such a combination involves applying a known data-feeding technique to a known CNN accelerator architecture to yield the predictable result of improved cycle-level utilization.
As to claim 24, Munteanu-Liu as modified by Feinberg teaches the method of claim 23,
wherein the plurality of registers includes N registers. (See Liu paragraph [0104] "register set 230 includes four data selectors 230-M.sub.1, 230-M.sub.2, 230-M.sub.3 and 230-M.sub.4, and four registers 230-X.sub.1, 230-X.sub.2, 230-X.sub.3 and 230-X.sub.4, arranged in a shift-loop.")
Examiner Note: The number of registers (four) in register set 230 corresponds to the number N of data values loaded per cycle (four column data values of a 4×4 IFM), satisfying the claim that the plurality of registers includes N registers.
As to claim 25, Munteanu-Liu as modified by Feinberg teaches the method of claim 23,
comprising passing, on each clock cycle, N−1 data values from the first register to a second register of the plurality of registers. (See Feinberg paragraph [0062] "Convolutional engine 708 may include a 2-D shift register with an array of data storage elements... Each of the arrows between pairs of data storage elements represents an electrical connection (i.e., may be implemented as a wire). For example, data storage element d.sub.1,1 (ref. num. 802) may be electrically coupled to storage element d.sub.2,1 (ref. num. 802) via electrical connection 804.", and See Feinberg paragraph [0071] "FIGS. 10B-10D depict a row-by-row loading of data values from the first input channel 702a into the 2-D shift register and a row-to-row shifting of the data values through the 2-D shift register.", and See Feinberg paragraph [0072] "Upon row n of horizontal stripe 902a being loaded into the second row of data storage elements (i.e., d.sub.2,1, d.sub.2,2, d.sub.2,3 and d.sub.2,4), the first row of convolver units (i.e., CU.sub.1,1, CU.sub.1,2, CU.sub.1,3 and CU.sub.1,4) corresponding to the second row of data storage elements may be activated.")
As to claim 26, Munteanu-Liu as modified by Feinberg teaches the method of claim 25,
comprising passing, on each clock cycle, N−2 data values from the second register to a third register of the plurality of registers. (See Feinberg paragraph [0062] "Convolutional engine 708 may include a 2-D shift register with an array of data storage elements... Each of the arrows between pairs of data storage elements represents an electrical connection (i.e., may be implemented as a wire). For example, data storage element d.sub.1,1 (ref. num. 802) may be electrically coupled to storage element d.sub.2,1 (ref. num. 802) via electrical connection 804.", and See Feinberg paragraph [0071] "FIGS. 10B-10D depict a row-by-row loading of data values from the first input channel 702a into the 2-D shift register and a row-to-row shifting of the data values through the 2-D shift register.", and See Feinberg paragraph [0072] "Upon row n of horizontal stripe 902a being loaded into the second row of data storage elements (i.e., d.sub.2,1, d.sub.2,2, d.sub.2,3 and d.sub.2,4), the first row of convolver units (i.e., CU.sub.1,1, CU.sub.1,2, CU.sub.1,3 and CU.sub.1,4) corresponding to the second row of data storage elements may be activated.")
As to claim 27, Munteanu-Liu as modified by Feinberg teaches the method of claim 23,
comprising shifting, on each clock cycle, a set of data locations of the first register that receive the N data values. (See Liu paragraph [0128] "The shift loop for register set 230 operates during states S2, S3, S4, S6, S7, S8 and S9. During operation in each of these states, the values stored in registers 230-X.sub.1, 230-X.sub.2, 230-X.sub.3, 230-X.sub.4 are shifted or rotated by one register in a closed loop manner")
Examiner Note: The shift-loop rotation changes which register positions contain newly loaded data versus shifted data, corresponding to shifting the set of data locations that receive the N data values. However, the specific pattern of shifting the starting slot within a single register is not explicitly taught; what is taught is an inter-register rotation.
As to claim 28, Munteanu-Liu as modified by Feinberg teaches the method of claim 23,
comprising, after a plurality of setup clock-cycles, successively reading a data set of the feature data from the registers on each clock cycle. (See Liu paragraph [0109] "State machine transition diagram 300 depicts nine unique states, i.e., S1, S2, S3, S4, S5, S6, S7, S8 and S9. Each state is associated with a particular processing cycle", and See Liu paragraph [0112] "Each state provides four register values, one value for each expanded activation sequence A1, A2, A3 and A4.")
Examiner Note: After the initial setup states (S1-S2 for loading data into registers), the MEU outputs data sets on each subsequent processing cycle, corresponding to successively reading data from the registers after setup clock-cycles.
As to claim 29, Munteanu-Liu as modified by Feinberg teaches the method of claim 23,
comprising passing a non-overlapping portion of the feature data from the memory to the first register on each clock cycle. (See Liu paragraph [0132] "During operation in state S1, controller 210 cooperates with controller 172 to store first column data of IFM 83 into registers 230-X.sub.1, 230-X.sub.1, 230-X.sub.1 and 230-X.sub.1 of register set 230.", and See Liu paragraph [0138] "State S5 operates in the same manner as state S1, loading the third column data of IFM 83 into registers 230-X.sub.1, 230-X.sub.2, 230-X.sub.3 and 230-X.sub.4 of register set 230, operating the shift loop of register set 240, and outputting the appropriate elements of each expanded activation sequence A.sub.1, A.sub.2, A.sub.3 and A.sub.4.")
Examiner Note: Each loading state loads a different column of IFM data (first column in S1, second column in S2, third column in S5, fourth column in S8), each column being a non-overlapping portion of the feature data.
Claim(s) 30-32, and 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Munteanu et al.(US 20200126178 A1) in view of Liu et al. (US 20210390367 A1) and Tuominen et al. (US 20090031089 A1).
As to claim 30, Munteanu as modified by Liu teaches the method of claim 21,
Munteanu-Liu does not explicitly teach "wherein storing the feature data in a memory of the internal storage includes storing, with a write transformer unit of the internal storage, the feature data in the memory of in the internal storage by at least partially transposing the rows and columns of the feature data"
However, Tuominen teaches:
wherein storing the feature data in a memory of the internal storage includes storing, with a write transformer unit of the internal storage, the feature data in the memory of in the internal storage by at least partially transposing the rows and columns of the feature data. (See Tuominen paragraph [0059] "A write address and word select offset logic 220 is coupled to the word select multiplexer 210 and the write address input WA 232 of the DPRAM component 230. The write address and word select offset logic 220 controls the operation of the word select multiplexer 210, which is configured to select one data word of the eight concatenated data words driven by the word combiner 300 on the 8-word wide concatenated bus 301.", and see Tuominen paragraph [0086] "the data words representing the matrix elements ye;ij driven on the concatenated bus 301 are stored in accordance with following writing scheme")
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the combined system of Munteanu and Liu to further incorporate the write-side transposition scheme of Tuominen, in which a write address and word select offset logic functions as a write transformer unit that stores incoming data with a transposed addressing pattern, because Tuominen teaches that its "write address and word select offset logic 220 controls the operation of the word select multiplexer 210, which is configured to select one data word of the eight concatenated data words" and "further controls into which storage cell of the DPRAM component 230 the one selected data word is written" (Tuominen paragraph [0059]), thereby achieving a storage arrangement in which row-oriented input data is distributed across memory blocks such that column-oriented readout can be performed efficiently without dead cycles, since "the transpose memory circuit is configured to receive the input matrix and to output the input matrix in transposed form in subsequent cycles without any dead cycles interposed between them" (Tuominen claim 1). A person of ordinary skill would have been motivated to apply Tuominen's transposed write addressing to Munteanu's image cache because Munteanu's convolution engine requires data organized in N×M windows for efficient single-cycle access (Munteanu paragraph [0060] "producing rectangular windows of N×M pixels for use within the convolution engine 32 in as few clock cycles as possible and preferably in a single clock cycle"), and Tuominen's write-time transposition directly enables such column-aligned data retrieval from row-stored memory, yielding the predictable benefit of faster and stall-free data reorganization.
As to claim 31, Munteanu-Liu as modified by Tuominen teaches the method of claim 30,
wherein passing, with a read transformer unit of the internal storage, the feature data to the plurality of registers includes passing at least a portion of each of N rows of the memory to a respective register from the plurality of registers. (See Liu paragraph [0132] "During operation in state S1, controller 210 cooperates with controller 172 to store first column data of IFM 83 into registers 230-X.sub.1, 230-X.sub.1, 230-X.sub.1 and 230-X.sub.1 of register set 230.")
Examiner Note: In Liu, data from each row of the local memory is read and sent to a respective register within a register set, with each register receiving a portion of data from a corresponding memory row, though the data is organized as columns of the IFM rather than physical memory rows.
As to claim 32, Munteanu-Liu as modified by Tuominen teaches the method of claim 31,
wherein passing the feature data from the registers includes reading a data set from each register on successive clock cycles in an alternating manner. (See Liu paragraph [0112] "Each state provides four register values, one value for each expanded activation sequence A.sub.1, A.sub.2, A.sub.3 and A.sub.4.", and see Liu paragraph [0113] "During operation in state S1, controller 210 sends a control signal to output data selector 250 to select register 230-X.sub.1 to output a first element of A.sub.1, select register 230-X.sub.2 to output a first element of A.sub.2, select register 240-Y.sub.1 to output an element of A.sub.3, and select register 240-Y.sub.2 to output an element of A.sub.4.", and see Liu paragraph [0116] " During operation in state S4, controller 210 sends a control signal to output data selector 250 to select register 240-Y.sub.3 to output the fourth element of A.sub.1, select register 240-Y.sub.4 to output the fourth element of A.sub.2, select register 240-Y.sub.1 to output the third element of A.sub.3, and select register 240-Y.sub.2 to output the third element of A.sub.4.")
Examiner Note: The output data selector alternates between reading from registers of register set 230 (states S1-S3) and register set 240 (states S4-S6) in successive clock cycles, corresponding to reading data sets from each register on successive clock cycles in an alternating manner.
As to claim 39, Munteanu-Liu as modified by Tuominen teaches the device of claim 38, wherein the internal storage includes a write transformer unit configured to write the feature data into the memory with a write address pattern based on a configuration of the hardware accelerator. (See Tuominen paragraph [0059]: "A write address and word select offset logic 220 is coupled to the word select multiplexer 210 and the write address input WA 232 of the DPRAM component 230. The write address and word select offset logic 220 controls the operation of the word select multiplexer 210, which is configured to select one data word of the eight concatenated data words driven by the word combiner 300 on the 8-word wide concatenated bus 301. The write address and word select offset logic 220 further controls into which storage cell of the DPRAM component 230 the one selected data word is written.")
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the device of Munteanu as combined with Liu to further include a write transformer unit configured to write feature data into the internal storage memory with a write address pattern based on a configuration of the hardware accelerator, as taught by Tuominen, because Tuominen discloses that its write address and word select offset logic generates individualized write addresses and word select offsets for each memory block based on a counter value and block identifier (Tuominen paragraph [0059] "The write address and word select offset logic 220 further controls into which storage cell of the DPRAM component 230 the one selected data word is written"), and that this write-time data distribution across memory blocks is specifically designed to enable efficient transposed readout for downstream processing (Tuominen paragraph [0086] "the data words representing the matrix elements ye;ij driven on the concatenated bus 301 are stored in accordance with following writing scheme"). A person of ordinary skill would have recognized that incorporating such a configurable write addressing unit into Munteanu's image cache, which already employs a cache control block that "calculates input data de-multiplexer and output data multiplexer selection signals" and "transforms the x, y, w and h inputs into Address (ADDR) and Byte (Pixel) Enable (BE) control signals for each of the four SRAMs" (Munteanu paragraph [0072]-[0073]), would provide the additional capability of adapting the write pattern to the specific data access requirements of the downstream convolution accelerator, with a reasonable expectation of success given the compatible SRAM-based architectures of both references.
Claim(s) 33-37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Munteanu et al.(US 20200126178 A1) in view of Tuominen et al. (US 20090031089 A1).
As to claim 33, Munteanu teaches a method, comprising:
receiving, at a neural network, feature data arranged in rows and columns from a memory external to the neural network; (See Munteanu paragraph [0035] "The CNN Engine 30 connects to a system bus 42 and can access main (DRAM) memory 40 into which images acquired by the system are written.", and see Munteanu paragraph [0064] "At each (x, y) SRAM address coordinate, a block of 4×4 pixels is stored.")
Examiner Note: Image data in DRAM 40 is inherently arranged in rows and columns (pixel rows and columns of the image).
passing the feature data to an internal storage of the neural network; (See Munteanu paragraph [0044] "the image cache 31 is initially loaded with an input image/map from DRAM 40.")
passing the feature data from the memory of the internal storage to a hardware accelerator; and (See Munteanu paragraph [0059] "the convolution engine 32 can process windows of N×M pixels provided by the image cache 31 each clock cycle.")
generating first transformed feature data by processing the feature data with the hardware accelerator. (See Munteanu paragraph [0057] "In the feature extraction mode the weights needed to generate a current output map are read from DRAM 40 by the read controller 38; and the input image/map is initially read by the read controller 36. The input data for each output map pixel and the weights are presented to the convolution engine 32.")
Munteanu does not explicitly teach "storing, with a write transformer unit of the internal storage, the feature data in a memory of in the internal storage by at least partially transposing the rows and columns of the feature data."
However, Tuominen teaches:
storing, with a write transformer unit of the internal storage, the feature data in a memory of in the internal storage by at least partially transposing the rows and columns of the feature data; (See Tuominen paragraph [0059] "A write address and word select offset logic 220 is coupled to the word select multiplexer 210 and the write address input WA 232 of the DPRAM component 230. The write address and word select offset logic 220 controls the operation of the word select multiplexer 210, which is configured to select one data word of the eight concatenated data words driven by the word combiner 300 on the 8-word wide concatenated bus 301. The write address and word select offset logic 220 further controls into which storage cell of the DPRAM component 230 the one selected data word is written.")
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the CNN Engine of Munteanu, which receives image data arranged in rows and columns from external DRAM into an internal image cache and provides the data to a convolution engine for generating transformed feature data, to further incorporate the write-side transposition technique of Tuominen, in which a write address and word select offset logic stores incoming row-oriented matrix data into memory with a transposed addressing pattern, because Tuominen teaches that this approach enables fully pipelined matrix transposition where "the transpose memory (TRAM) 105 accepts an 8-word data vector for being stored and provides an 8-word data vector for being read out on each clock cycle" and operates "in a streaming mode without pipeline stalls or any other undesired interrupts" (Tuominen paragraph [0052]). A person of ordinary skill would have been motivated to apply Tuominen's transposed write storage to Munteanu's image cache because Munteanu acknowledges the need for efficient single-cycle access to rectangular pixel windows for convolution operations (Munteanu paragraph [0060] "the image cache 31 is arranged to accelerate processing by producing rectangular windows of N×M pixels for use within the convolution engine 32 in as few clock cycles as possible and preferably in a single clock cycle"), and Tuominen's write-time transposition would directly facilitate such access by pre-arranging column data into row-accessible positions at write time, eliminating the need for costly multi-cycle read reorganization and yielding the predictable result of improved data throughput between the cache and the convolution engine, with a reasonable expectation of success given that both references employ SRAM-based local storage architectures in the same general field of hardware-accelerated data processing.
As to claim 34, Munteanu as modified by Tuominen teaches the method of claim 33,
wherein passing the feature data from the memory of the internal storage includes reading, on each clock cycle, a data set entirely from a single row of the memory. (See Munteanu paragraph [0060] "the image cache 31 is arranged to accelerate processing by producing rectangular windows of N×M pixels for use within the convolution engine 32 in as few clock cycles as possible and preferably in a single clock cycle.", and see Munteanu paragraph [0064] "This arrangement allows reading and writing of blocks of pixels in only one clock cycle")
As to claim 35, Munteanu as modified by Tuominen teaches the method of claim 33,
wherein the at least partially transposing includes storing multiple columns of the feature data in a single row of the memory. (See Tuominen paragraph [0076] "The data words representing the matrix elements ye;00, . . . , ye;07 are selected by the DPRAM blocks h=0 to 7 in accordance with their original sequence order.", and see Tuominen paragraph [0081] " the data words representing the matrix elements ye;35, ye;36, ye;37, ye;30, . . . , ye;34 have been written into the DPRAM component of the DPRAM blocks 0 to 7 is schematically illustrated in FIG. 5d.")
As to claim 36, Munteanu as modified by Tuominen teaches the method of claim 33,
wherein storing the feature data in the memory includes storing multiple pixels of a column of the feature data in a single data location of the memory. (See Munteanu paragraph [0064] "At each (x, y) SRAM address coordinate, a block of 4×4 pixels is stored. So, at address 0x00 of SRAM2, a block of 4×4 pixel values from index 40 to 73 are stored.")
As to claim 37, Munteanu as modified by Tuominen teaches the method of claim 33, wherein the hardware accelerator is a convolution accelerator. (See Munteanu paragraph [0057] "In the feature extraction mode the weights needed to generate a current output map are read from DRAM 40 by the read controller 38; and the input image/map is initially read by the read controller 36. The input data for each output map pixel and the weights are presented to the convolution engine 32.")
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABDULLAH K ABOUD whose telephone number is (571)272-0025. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen, can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABDULLAH KHALED ABOUD/ Examiner, Art Unit 2121
/Li B. Zhen/ Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights