DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Action is Non-Final and is in response to the claims filed 02/10/2023. Claims 1-15 are currently pending, of which claims 1-15 are currently rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 14 is rejected under 35 U.S.C. 101 because a computer-readable medium being a computer-readable medium storing instructions to be executed by a processing unit would normally be considered statutory unless the specification defines "computer-readable medium storing instructions" as including transient media such as signals, carries waves, transmissions, optical waves, transmission media or other media incapable of being touched or perceived absent the non-transitory medium through which they are conveyed.
Claim 14 is not limited to non-transitory embodiments. Specifically, in view of the specification (¶00017), the computer-readable medium storing instructions is not limited to non-transitory embodiments. Instead, the specification repeats the claim language for describing the computer-readable medium storing instructions. Therefore, the claim is not limited to statutory subject matter, hence Claim 14 is non-statutory.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Arash Azizimazreah NPL: “Flexible On-chip Memory Architecture for DCNN Accelerators” (cited on IDS 03/14/2023), hereinafter “Arash”, in view of Zhu et al. (U.S. Patent Application Publication No.: US 20220414423 A1), hereinafter “Zhu”, further in view of Xu et al. (U.S. Patent Application Publication No.: US 20220130142 A1), hereinafter “Xu”.
Regarding Claim 1, Arash teaches:
A method performed by a processing unit for generating an output feature map (Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines outputting output feature maps (OFMs) to output buffers; Section 2.3), the processing unit comprising an input feature map storage configured to store input feature map blocks (Fig. 1, e.g., shows input buffers storing Tn words; Section 2.2 Tiling, e.g., Tn is the tiling factor on N Input Feature Maps (IFMs)), the input feature map storage being readable by the processing unit to generate output feature map blocks (Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines outputting output feature maps to output buffers by processing IFMs), the method comprising:
… wherein the input feature map blocks stored in the input feature map storage … form a subset of a plurality of input feature map blocks that are required to generate each output feature map block (Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines outputting output feature maps to output buffers by processing Tn words (IFMs) stored in memory banks in input buffers; Section 2.2 Tiling, e.g., Tn is the tiling factor on N Input Feature Maps (IFMs));
reading a first input feature map block stored in the input feature map storage during reading of a first sequence of input feature map blocks used to generate partial computations for a first output feature map block (Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines each processing Tn words (IFMs) stored in memory banks in input buffers, and outputting output feature maps to output buffers; Section 2.2 Tiling, e.g., Tn is the tiling factor on N Input Feature Maps (IFMs)); and
reading a second time the first input feature map block stored in the input feature map storage during reading of [the first sequence] of input feature map blocks used to generate partial computations for a second output feature map block (Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines each processing Tn words (IFMs) stored in memory banks in input buffers, and outputting output feature maps to output buffers; Section 2.2 Tiling), … .
Arash does not teach:
sequentially loading input feature map blocks into the input feature map storage, wherein the input feature map blocks stored in the input feature map storage at a given time during the sequential loading form a subset of a plurality of input feature map blocks that are required to generate each output feature map block;
…
reading a second time the first input feature map block stored in the input feature map storage during reading of a second sequence of input feature map blocks used to generate partial computations for a second output feature map block, wherein the first sequence is different from the second sequence.
However, Zhu teaches:
sequentially loading input feature map blocks into the input feature map storage, wherein the input feature map blocks stored in the input feature map storage at a given time during the sequential loading form a subset of a plurality of input feature map blocks that … (Abstract, e.g., sequentially stores input feature maps into cache sub-blocks according to a loading length; ¶0045, e.g., kernels are loaded at a time of loading);
Arash does not teach how input feature maps are loaded to the input buffers. Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to combine the sequential loading of input feature maps at a load time as taught by Zhu with the storing of input feature maps in input buffers as taught by Arash. One would have been motivated to combine these references because both references disclose convolutional neural networks processing input feature maps, and Zhu enhances the model of Arash by allowing for input feature maps to be loaded to the input buffers.
Arash in view of Zhu do not teach:
reading a second time the first input feature map block stored in the input feature map storage during reading of a second sequence of input feature map blocks used to generate partial computations for a second output feature map block, wherein the first sequence is different from the second sequence.
However, in the same field of endeavor, Xu teaches traversing a feature map in reverse for a node processing to generate output feature maps. Xu explains “As shown in FIG. 11, the node 0 outputs a first feature map, some feature maps of the first feature map are processed by using the to-be-selected operation, to obtain a second feature map, and the second feature map may be input to the node 1 and continue to be processed after a channel sequence is reversed.” (Xu: ¶0249 and Fig. 11)
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to combine reversing of channel sequence for different nodes as taught by Xu with the input feature map processing as taught by Arash in view of Zhu. One would have been motivated to combine these references because both references disclose processing of feature maps in convolutional neural networks, and Xu enhances the model of Arash in view of Zhu because “This may increase randomness of input data, and avoid overfitting of the finally obtained target neural network as much as possible.” (Xu: ¶0040). Combination would cause for a first sequence to be different from the second sequence reading feature maps in reverse order for vector-dot-product processing.
Regarding Claim 2, Arash in view of Zhu in view of Xu teach:
The method according to claim 1, wherein the first input feature map block is a final input feature map block of the first sequence, and the first input feature map block is a first input feature map block of the second sequence (Xu: ¶0224, e.g., node 0 and 1 respectively process input data; ¶0249, e.g., feature maps are reversed for node 1; Fig. 12, e.g., shows the reversing of feature maps).
The motivation to combine provided with respect to claim 1 applies equally to claim 2.
Regarding Claim 3, Arash in view of Zhu in view of Xu teach:
The method according to claim 1, wherein the first sequence is a linear sequence of input feature map blocks in a first direction along a channel dimension of the input feature map (Xu: Fig. 12, e.g., shows feature maps before reversing; ¶0262, e.g., node 0 processes c channels (linear sequence)).
The motivation to combine provided with respect to claim 1 applies equally to claim 3.
Regarding Claim 4, Arash in view of Zhu in view of Xu teach:
The method according to claim 3, wherein the second sequence is the reverse of the first sequence (Xu: ¶0249, e.g., feature maps are reversed for node 1).
The motivation to combine provided with respect to claim 1 applies equally to claim 4.
Regarding Claim 5, Arash in view of Zhu in view of Xu teach:
The method according to claim 1, wherein the processing unit transfers the input feature map blocks from the input feature map storage to dot product units, and the dot product units process the input feature map blocks for generating the output feature map blocks (Arash: Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines each processing Tn words (IFMs) stored in memory banks in input buffers, and outputting output feature maps to output buffers).
Regarding Claim 6, Arash in view of Zhu in view of Xu teach:
The method according to claim 5, wherein:
the input feature map blocks each comprise at least one input feature map channel (Arash: Fig. 1; Section 2.2 Tiling, e.g., Tn is the tiling factor on N Input Feature Maps (IFMs); Xu: Fig. 12, e.g., shows channels of feature maps),
the output feature map blocks each comprise at least one output feature map channel (Arash: Fig. 1; Section 2.2 Tiling, e.g., Tm is the tiling factor on M Output Feature Maps (OFMs); Xu: Fig. 12, e.g., shows channels of feature maps),
a set of weights comprises rows of weights, each row of weights corresponding to an output feature map channel and weight values in each row of weights corresponding to the input feature map channels (Arash: Fig. 1, e.g., shows weights for each Vector-dot-product engine corresponding to each output Tm (OFMs) and input Tn (IFMs); Section 2.2 Tiling), and the method comprises:
generating a partial computation for a given output feature map block using a given input feature map block by calculating, by the dot product units, for each output feature map channel of the given output feature map block, a dot product between a partial row of weights corresponding to each output feature map channel and an input feature map vector, the input feature map vector comprising the input feature map elements for each input feature map channel of the given input feature map block (Arash: Fig. 1, e.g., shows weights for each Vector-dot-product engine (dot product unit) corresponding to each output Tm (OFMs) (partial computations) and input Tn (IFMs); Section 2.2 Tiling; Xu: Fig. 12, e.g., shows channels of feature maps); and
generating the given output feature map block comprises, for each output feature map channel of the given output feature map block, summing the dot products generated for the output feature map channel (Arash: Fig. 1, e.g., shows adders adding data from output buffers and the outputs of the Vector-dot-product engines; Section 2.3 On-chip Memory Sub-System, e.g., Tm pixels represent partial sums; Xu: Fig. 12, e.g., shows channels of feature maps).
The motivation to combine provided with respect to claim 1 applies equally to claim 6.
Regarding Claim 7, Arash in view of Zhu in view of Xu teach:
The method according to claim 1, wherein the input feature map storage is able to store a predetermined number of input feature map blocks (Arash: Fig. 1, e.g., shows input buffers storing data on a number of memory banks).
Regarding Claim 8, Arash in view of Zhu in view of Xu teach:
The method according to claim 7, wherein the method comprises reusing the predetermined number of input feature map blocks without reloading the predetermined number of input feature map blocks into the input feature map storage to generate partial computations for the second output feature map block (Arash: Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines each processing Tn words (IFMs) stored in memory banks in input buffers, and outputting output feature maps to output buffers; Section 2.2 Tiling).
Regarding Claim 13, Arash in view of Zhu in view of Xu teach:
A method according to claim 1, wherein sequentially loading the input feature map blocks into the input feature map storage comprises transferring the input feature map blocks from an external storage that is external to the processing unit to the input feature map storage (Zhu: Abstract, e.g., sequentially stores input feature maps into cache sub-blocks according to a loading length; ¶0054, e.g., input feature maps are loaded from off-chip input feature map storage).
The motivation to combine provided with respect to claim 1 applies equally to claim 13.
Regarding Claim 14, it is an apparatus claim version of the method claim 1. It is rejected for the same reasons as claim 1.
Regarding Claim 15, it is a media claim version of the method claim 1. It is rejected for the same reasons as claim 1.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Arash in view of Zhu in view of Xu, further in view of Kim et al. (U.S. Patent Application Publication No.: US 20180189643 A1), hereinafter “Kim”.
Regarding Claim 9, Arash in view of Zhu in view of Xu teach:
The method according to claim 1, wherein the input feature map storage is controlled to operate as a … buffer (Arash: Fig. 1, e.g., shows input buffers).
Arash in view of Zhu in view of Xu do not teach:
wherein the input feature map storage is controlled to operate as a first-in first-out buffer.
However, Kim teaches:
wherein the input feature map storage is controlled to operate as a first-in first-out buffer (¶0015, e.g., First-in, first out memory stores input feature maps).
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the input buffers as taught by Arash to be first-in, first-out memory as taught by Kim. One would have been motivated to combine these references because both references disclose storing input feature maps for convolution operations, and Kim enhances the model of Arash in view of Zhu in view of Xu by allowing to read input data as a first-in first-out manner to determine DMA read complete when the last data is read. See Kim: ¶0078.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Arash in view of Zhu in view of Xu in view of Kim, further in view of Lo et al. (U.S. Patent Application No.: US 20210173648 A1), hereinafter “Lo”.
Regarding Claim 10, Arash in view of Zhu in view of Xu in view of Kim teach:
The method according to claim 9, wherein:
the first input feature map block is both a final input feature map block of the first sequence and a first input feature map block of the second sequence (Xu: ¶0224, e.g., node 0 and 1 respectively process input data; ¶0249, e.g., feature maps are reversed for node 1; Fig. 12, e.g., shows the reversing of feature maps), and
the processing unit controls … the sequence in which input feature map blocks are used to generate partial computations for the second output feature map block (Xu: ¶0224, e.g., node 0 and 1 respectively process input data; ¶0249, e.g., feature maps are reversed for node 1; Fig. 12, e.g., shows the reversing of feature maps; Arash: Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines each processing Tn words (IFMs) stored in memory banks in input buffers, and outputting output feature maps to output buffers; Section 2.2 Tiling).
Arash in view of Zhu in view of Xu in view of Kim do not teach:
the processing unit controls an address pointer of the input feature map storage to control the sequence in which input feature map blocks are used to generate partial computations for the second output feature map block.
However, in the same field of endeavor, Lo teaches using an input pointer to point to memory addresses to read/write feature maps for processing. Lo explains “input pointer 331 points to the third memory address of the scratchpad memory 1 and makes the nth layer output feature maps stored therein serve as the to-be-processed data for the (n+1)th layer” (Lo: ¶0044). Lo further explains “The neural network accelerator is electrically coupled to the processor core and the scratchpad memory, and is configured to issue accelerator-side read/write instructions that conform with the memory interface to access the scratchpad memory for acquiring the to-be-processed data” (Lo: ¶0009)
Arash in view of Zhu in view of Xu in view of Kim do not disclose how feature maps are being accessed for processing. Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the Convolution layer processor as taught by Arash in view of Zhu in view of Xu in view of Kim to include an address pointer to read/write data to be processed as taught by Lo. One would have been motivated to combine these references because both references disclose processing of feature maps for convolutional neural networks, and Lo enhances the model of Arash in view of Zhu in view of Xu in view of Kim by allowing for feature maps to be accessed for dot-product processing.
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Arash in view of Zhu in view of Xu in view of Lo, further in view of Wang et al. (U.S. Patent Application Publication No.: US 20220365782 A1), hereinafter “Wang”.
Regarding Claim 11, Arash in view of Zhu in view of Xu in view of Lo teach:
The method according to claim 1, wherein the processing unit controls a read pointer and a … release input feature map blocks and the read pointer … is controlled so that the first input feature map block can be reused (Arash: Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines each processing Tn words (IFMs) stored in memory banks in input buffers, and outputting output feature maps to output buffers; Lo: ¶0009, e.g., read/write instructions are used to acquire to-be-processed data; ¶0044, e.g., pointers point to address in memory to acquire to-be-processed data).
The motivation to combine provided with respect to claim 10 applies equally to claim 11.
Arash in view of Zhu in view of Xu in view of Lo do not teach:
The method according to claim 1, wherein the processing unit controls a read pointer and a release signal to release input feature map blocks and the read pointer and release signal are controlled so that the first input feature map block can be reused.
However, Wang teaches using a feature-map-ready signal to determine if a feature map is ready to release a buffer to load feature maps. Wang explains “When the feature-map-ready signal is enabled for the first time, the Fmap feeder reads the data into level-0 buffer, and after the computation units finish processing the data in level-0 buffer, the Fmap feeder may push the data values in the level-0 buffer to the level-1 buffer and release the level-0 buffer for loading next block of data when the feature-map-ready signal is enabled again.” (Wang: ¶0059)
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to modify the Convolution layer processor as taught by Arash in view of Zhu in view of Xu in view of Kim to include the feature-map-ready signal as taught by Wang. One would have been motivated to combine these references because both references disclose processing of feature maps for convolutional neural networks, and Wang enhances the model of Arash in view of Zhu in view of Xu in view of Lo by “indicating that the feature map is ready to be read by the array of computation units for calculation” (Wang: ¶0057).
Regarding Claim 12, Arash in view of Zhu in view of Xu in view of Lo in view of Wang teach:
The method according to claim 11, wherein the processing controls a write pointer to control writing of input feature map blocks to the input feature map storage and the read pointer, release signal and write pointer are controlled to prevent writing over the first input feature map block in the input feature map storage until it has been reused to generate a partial computation for the second output feature map block (Arash: Fig. 1, e.g., shows convolution layer processor including Vector-dot-product engines each processing Tn words (IFMs) stored in memory banks in input buffers, and outputting output feature maps to output buffers; Lo: ¶0009, e.g., read/write instructions are used to acquire to-be-processed data; ¶0044, e.g., pointers point to address in memory to acquire to-be-processed data; Wang: ¶0057, e.g., indicates using a feature-map-ready signal if the feature map is ready to be read for processing).
The motivation to combine provided with respect to claim 11 applies equally to claim 12.
Prio Art Made of Record
NPL: “Arm® Ethos™-U65 NPU Revision: r0p0 Technical reference manual, Chapter 2 Functional description” – teaches a Functional blocks diagram that uses a MAC unit, Weight decoder, and DMA controller. See Figure 2-3. This is pertinent to figure 2 of the instant application showing Dot product units, DMA, and Weight decoder.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARLOS H DE LA GARZA whose telephone number is (571)272-0474. The examiner can normally be reached Monday-Friday 9:30AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Caldwell can be reached at (571) 272-3702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.H.D./
Carlos H. De La GarzaExaminer, Art Unit 2182 (571)272-0474
/ANDREW CALDWELL/Supervisory Patent Examiner, Art Unit 2182
Prosecution Insights