DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1 and 4-15 are presented for examination.
Response to Amendment
Applicant’s arguments with respect to the interpretation of the claims under 35 USC § 112(f) are persuasive. Therefore, that interpretation is withdrawn.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. The present application is a continuation application of international patent application no. PCT/KR2021/015706, filed on 11/02/2021, which claims foreign priority to Korean patent application no. KR10-2020-0147081, filed on 11/05/2020.
The examiner acknowledges that a certified copy of Korean patent application No. KR10-2020-0147081 has been retrieved (on 5/23/2023, in Korean), as required by 37 CFR 1.55. The examiner notes that a translation of Korean patent application No. KR10-2020-0147081 does not appear to have been furnished to date.
Claim Objections
Claim 14 is objected to because of the following informalities: there appears to be a superfluous space between “mapping” and the comma. Appropriate correction is required.
Claim Rejections - 35 USC § 103
Claims 1, 14, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publication No. US 20170344876 A1 (Brothers) in view of Lee (“UNPU: An Energy-Efficient Deep Neural Network Accelerator with Fully Variable Weight Bit Precision”).
Claim 1.
Brothers discloses an electronic device, comprising (Para [0051] “an electronic device 500 that includes one or more integrated circuits (chips) forming a system” teaches an electronic device):
an input feature map transformer configured to transform an input feature map (IFM) to a Winograd domain (Para [0020] “In each of the input feature maps, multiple patches may be transformed into, for example, the Winograd domain” teaches transformed an input feature map to Winograd domain);
a weight kernel transformer configured to transform a weight kernel to the Winograd domain (Para [0026] “each IDP unit reads and transforms a single input feature map, but 16 different transformed weight kernels are applied to each input map, one kernel per output feature map. The IDP unit transforms the 16 patches of input data into, for example, the Winograd domain” teaches a weight kernels transform to the Winograd domain);
a transformation data processor configured to: map … a plurality of feature groups created by grouping feature values in a plurality of channels of the transformed input feature map and a plurality of weight value groups created by grouping weight values in a plurality of channels of the transformed weight kernel, to at least one type of MAC unit from among a first type MAC unit, a second type MAC unit, and a third type MAC unit included in a plurality of types of multiply-accumulate (MAC) units in a computation unit being configured to perform a MAC operation between a weight value group of the plurality of weight value groups and a feature value group of the plurality of feature groups (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results… each of the eight inputs and their corresponding weights coming from eight different input feature maps might be for element in position (1,3) in a given cycle. Each IDP unit generates processing requests for different input feature maps and, since the corresponding weight kernels have 0 weights at different positions in the kernels, the requests generated by the input fetch units may be for processing of different output elements” teaches multiply-accumulate units (each being a separate type) that perform computations, creating eight different input feature maps (corresponds to the plurality of feature groups) and a set of eight weights (corresponds to the plurality of weight value groups));
a computation data processor configured to collect MAC operation results from the computation unit (Para [0026] “The IDP unit then sends 16 requests to 16 different request-assembly units, each corresponding to a different output feature map. Each request assembly unit feeds a different set of 16 MAUs, Each of the 16 request-assembly units and corresponding 16 MAUs act independently of the others, each generating 16 patches of one output map” teaches collect the result of the multiply-accumulate units);
and an inverse transformer configured to perform an inverse Winograd transform on a result output according to the collected MAC operation results from the computation data processor to thereby generate an output feature map (OFM) useable for performing a convolution operation (Para [0026] “The IDP unit then sends 16 requests to 16 different request-assembly units, each corresponding to a different output feature map. Each request assembly unit feeds a different set of 16 MAUs,. Each of the 16 request-assembly units and corresponding 16 MAUs act independently of the others, each generating 16 patches of one output map”; Para [0036] “The subject matter disclosed herein includes multiple IDPs 102a-102n in which the respective elements of a transformed feature-map data patch and the weight values in the corresponding transformed weight kernel are multiplied together to form a convolved matrix in the Winograd domain. The elements in the n convolved matrices are respectively summed and the result, indicated at 103, is inverse Winograd transformed to form a 2×2 output feature map patch” and Para [0029] “each output matrix represents a 2×2 patch in the output map—four elements—and it takes four multiply operations to apply the convolution with the Winograd scheme while skipping 0-valued weights, the overall computation per output element is just one multiply” teaches an inverse Winograd transform on the MAU result process to output feature map for performing a convolution operation).
Brothers appears not to disclose explicitly the further limitations of the claim. However, Lee discloses map[ping], based on a pre-generated mapping table, a plurality of feature groups (A. Detail Architecture of the Unified DNN Core & Page 176 “Four LUT bundles are included in the LBPE, and each LUT bundle is used for MAC operations by accessing LUT in it. The LBPE generates partial sums of DNN output feature map by accumulating the partial sums obtained from the four LUT bundles” teaches generating a feature map and weight kernel of the MAUs using a lookup table (LUT (see, Abstract)), i.e., a pre-generated mapping table) ….
Lee and the instant application are analogous art because they are both directed to using a hardware architecture to accelerate deep learning through convolutional and multiply accumulate processing.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Lee into the disclosed invention of Brothers.
One of ordinary skill in the arts would have been motivated to make this modification because of the following: the lookup table (LUT)-based bit-serial processing element (LBPE) provides “energy consumption reduction compared to the conventional fixed-point multiply-and-accumulate (MAC) array” while supporting efficient processing of non-zero transformed weights, as suggested by Lee (Lee, Abstract, Page 173).
Claim 14.
Brothers discloses an operation method of an electronic device, comprising (Para [0051] “an electronic device 500 that includes one or more integrated circuits (chips) forming a system” teaches an electronic device):
transforming an input feature map (IFM) to a Winograd domain (Para [0020] “In each of the input feature maps, multiple patches may be transformed into, for example, the Winograd domain” teaches transforming input feature maps to the Winograd domain);
transforming a weight kernel to the Winograd domain (Para [0026] “each IDP unit reads and transforms a single input feature map, but 16 different transformed weight kernels are applied to each input map, one kernel per output feature map. The IDP unit transforms the 16 patches of input data into, for example, the Winograd domain” teaches a weight kernels transform to the Winograd domain);
creating a plurality of feature value groups by grouping feature values at same coordinates in a plurality of channels of the transformed input feature map (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results…each of the eight inputs and their corresponding weights coming from eight different input feature maps might be for element in position (1,3) in a given cycle” and Para [0034] “parallelizing in the input channel dimension by having multiple IDPs each operate on different input maps in parallel (for example, eight input maps fed into the each MAU after reordering in the RAU), it is also possible and advantageous to parallelize in the output channel dimension” teaches grouping eight different input feature maps (corresponds to feature values) to a same channel);
creating a plurality of weight value groups by grouping weight values at same coordinates in a plurality of channels of the transformed weight kernel (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results…each of the eight inputs and their corresponding weights coming from eight different input feature maps might be for element in position (1,3) in a given cycle” and Para [0034] “parallelizing in the input channel dimension by having multiple IDPs each operate on different input maps in parallel (for example, eight input maps fed into the each MAU after reordering in the RAU), it is also possible and advantageous to parallelize in the output channel dimension” teaches grouping eight different set of eight weights values to a same channel);
mapping … the plurality of feature value groups and the plurality of weight value groups to at least one type of MAC unit from among a first type MAC unit, a second type MAC unit, and a third type MAC unit included in a plurality of types of multiply-accumulate (MAC) units included in the electronic device (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results… each of the eight inputs and their corresponding weights coming from eight different input feature maps might be for element in position (1,3) in a given cycle. Each IDP unit generates processing requests for different input feature maps and, since the corresponding weight kernels have 0 weights at different positions in the kernels, the requests generated by the input fetch units may be for processing of different output elements” teaches that the multiply-accumulate units perform computations, transform eight different input feature maps, and have a set of eight weights, as well as multiple MAUs);
outputting a MAC operation value by performing, a MAC operation, for the plurality of feature value groups with the plurality of weight value groups, respectively ((Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results” teaches outputting a multiply-accumulate unit performing an operation using a feature value and weight value of eight sets);
generating a transformed output feature map by collecting MAC operation results according to the outputting of the output MAC operation value (Para [0025] “The eight results are all added into 1-of-16 accumulator registers maintained by the IDP unit, each corresponding to one of the 16 elements of the output patch being computed by the MAU. Each of the 16 MAUs compute a different one of the 16 output patches being computed in parallel” and Para [0026] “16 output patches are processed in parallel to generate a portion of one output feature map” teaches generating a transformed output feature map by collecting 16 output of the multiply-accumulate units (MAU) operation value);
and performing an inverse Winograd transform on the generated transformed output feature map to thereby generate an output feature map (OFM) useable for performing a convolution operation (Para [0026] “The IDP unit then sends 16 requests to 16 different request-assembly units, each corresponding to a different output feature map. Each request assembly unit feeds a different set of 16 MAUs,. Each of the 16 request-assembly units and corresponding 16 MAUs act independently of the others, each generating 16 patches of one output map”; Para [0036] “The subject matter disclosed herein includes multiple IDPs 102a-102n in which the respective elements of a transformed feature-map data patch and the weight values in the corresponding transformed weight kernel are multiplied together to form a convolved matrix in the Winograd domain. The elements in the n convolved matrices are respectively summed and the result, indicated at 103, is inverse Winograd transformed to form a 2×2 output feature map patch” and Para [0029] “each output matrix represents a 2×2 patch in the output map—four elements—and it takes four multiply operations to apply the convolution with the Winograd scheme while skipping 0-valued weights, the overall computation per output element is just one multiply” teaches an inverse Winograd transform on the MAU result process to output a feature map for performing a convolution operation).
Brothers appears not to disclose explicitly the further limitations of the claim. However, Lee discloses mapping, based on a pre-generated mapping table, the plurality of feature groups (A. Detail Architecture of the Unified DNN Core & Page 176 “Four LUT bundles are included in the LBPE, and each LUT bundle is used for MAC operations by accessing LUT in it. The LBPE generates partial sums of DNN output feature map by accumulating the partial sums obtained from the four LUT bundles” teaches generating a feature map and weight kernel of the MAUs using a lookup table (LUT (see, Abstract)), i.e., a pre-generated mapping table) ….
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Lee into the disclosed invention of Brothers.
One of ordinary skill in the arts would have been motivated to make this modification because of the following: the lookup table (LUT)-based bit-serial processing element (LBPE) provides “energy consumption reduction compared to the conventional fixed-point multiply-and-accumulate (MAC) array” while supporting efficient processing of non-zero transformed weights, as suggested by Lee (Lee, Abstract, Page 173).
Claim 15.
Brothers discloses a non-transitory computer-readable recording medium having recorded thereon a program to execute a method by an electronic device, the method comprising (Para [0051] “an electronic device 500 that includes one or more integrated circuits (chips)…Electronic device 500 may be used in, but not limited to, a computing device, a personal digital assistant (PDA), a laptop computer, a mobile computer, a web tablet, a wireless phone, a cell phone, a smart phone, a digital music player, or a wireline or wireless electronic device. The electronic device 500 may include a controller 510, an input/output device 520 such as, but not limited to, a keypad, a keyboard, a display, a touch-screen display, a camera, and/or an image sensor, a memory 530” teaches an electronic device):
transforming an input feature map (IFM) to a Winograd domain (Para [0020] “In each of the input feature maps, multiple patches may be transformed into, for example, the Winograd domain” teaches transformed input feature maps to Winograd domain);
transforming a weight kernel to the Winograd domain (Para [0026] “each IDP unit reads and transforms a single input feature map, but 16 different transformed weight kernels are applied to each input map, one kernel per output feature map. The IDP unit transforms the 16 patches of input data into, for example, the Winograd domain” teaches a weight kernels transform to the Winograd domain);
grouping feature values at same coordinates in a plurality of channels of the transformed input feature map to create a plurality of feature value groups (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results…each of the eight inputs and their corresponding weights coming from eight different input feature maps might be for element in position (1,3) in a given cycle” and Para [0034] “parallelizing in the input channel dimension by having multiple IDPs each operate on different input maps in parallel (for example, eight input maps fed into the each MAU after reordering in the RAU), it is also possible and advantageous to parallelize in the output channel dimension” teaches grouping eight different input feature maps (corresponds to feature values) to same channel);
grouping weight values at same coordinates in a plurality of channels of the transformed weight kernel to create a plurality of weight value groups (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results…each of the eight inputs and their corresponding weights coming from eight different input feature maps might be for element in position (1,3) in a given cycle” and Para [0034] “parallelizing in the input channel dimension by having multiple IDPs each operate on different input maps in parallel (for example, eight input maps fed into the each MAU after reordering in the RAU), it is also possible and advantageous to parallelize in the output channel dimension” teaches grouping eight different set of eight weights values to same channel);
mapping … the plurality of feature value groups and the plurality of weight value groups to at least one type of MAC unit from among a first type of MAC unit, a second type MAC unit, and a third type MAC unit included in a plurality of types of multiply-accumulate (MAC) units included in the electronic device (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results… each of the eight inputs and their corresponding weights coming from eight different input feature maps might be for element in position (1,3) in a given cycle. Each IDP unit generates processing requests for different input feature maps and, since the corresponding weight kernels have 0 weights at different positions in the kernels, the requests generated by the input fetch units may be for processing of different output elements” teaches multiply-accumulate units performing computations, transforming eight different input feature maps, and a set of eight weights, as well as multiple MAUs);
outputting a MAC operation value by performing, a MAC operation, for the plurality of feature value groups, with the plurality of weight value groups, respectively (Para [0025] “An array of 16 multiply-accumulate units (MAU) are coupled to the request-assembly unit to process the requests. Each MAU takes as inputs eight sets of input values and a corresponding set of eight weights. In parallel, each of the eight inputs is multiplied by its corresponding weight to generate eight results” teaches outputting a multiply-accumulate units performing operation using a feature values and weight value of eight sets);
generating a transformed output feature map by collecting MAC operation results according to the outputting of the output MAC operation value (Para [0025] “The eight results are all added into 1-of-16 accumulator registers maintained by the IDP unit, each corresponding to one of the 16 elements of the output patch being computed by the MAU. Each of the 16 MAUs compute a different one of the 16 output patches being computed in parallel” and Para [0026] “16 output patches are processed in parallel to generate a portion of one output feature map” teaches generating a transformed output feature map by collecting 16 output of the multiply-accumulate units (MAU) operation value);
and performing an inverse Winograd transform on the generated transformed output feature map to thereby generate an output feature map (OFM) useable for performing a convolution operation (Para [0026] “The IDP unit then sends 16 requests to 16 different request-assembly units, each corresponding to a different output feature map. Each request assembly unit feeds a different set of 16 MAUs,. Each of the 16 request-assembly units and corresponding 16 MAUs act independently of the others, each generating 16 patches of one output map”; Para [0036] “The subject matter disclosed herein includes multiple IDPs 102a-102n in which the respective elements of a transformed feature-map data patch and the weight values in the corresponding transformed weight kernel are multiplied together to form a convolved matrix in the Winograd domain. The elements in the n convolved matrices are respectively summed and the result, indicated at 103, is inverse Winograd transformed to form a 2×2 output feature map patch” and Para [0029] “each output matrix represents a 2×2 patch in the output map—four elements—and it takes four multiply operations to apply the convolution with the Winograd scheme while skipping 0-valued weights, the overall computation per output element is just one multiply” teaches an inverse Winograd transform on the MAU result process to output feature map for performing a convolution operation).
Brothers appears not to disclose explicitly the further limitations of the claim. However, Lee discloses mapping, based on a pre-generated mapping table, the plurality of feature value groups (A. Detail Architecture of the Unified DNN Core & Page 176 “Four LUT bundles are included in the LBPE, and each LUT bundle is used for MAC operations by accessing LUT in it. The LBPE generates partial sums of DNN output feature map by accumulating the partial sums obtained from the four LUT bundles” teaches generating a feature map and weight kernel of the MAUs using a lookup table (LUT (see, Abstract)), i.e., a pre-generated mapping table) ….
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Lee into the disclosed invention of Brothers.
One of ordinary skill in the arts would have been motivated to make this modification because of the following: the lookup table (LUT)-based bit-serial processing element (LBPE) provides “energy consumption reduction compared to the conventional fixed-point multiply-and-accumulate (MAC) array” while supporting efficient processing of non-zero transformed weights, as suggested by Lee (Lee, Abstract, Page 173).
Allowable Subject Matter
Claims 4-13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. For example, with regard to dependent claims 4-13, the prior art of record does not anticipate, nor does it render obvious in any reasonable combination to one of ordinary skill in the art at the time of Applicants' invention, the combination of recited limitations of claims 4-13 and their base claim, independent claim 1, and their respective intervening claims.
Response to Arguments
Applicant's arguments filed June 3, 2026 (“Remarks”) have been fully considered but they are not persuasive.
Applicant argues that the multiple MAUs of Brothers is improperly mapped to the claimed three different types of MAC units because all of the MAUs in Brothers are of the same type, where “type” allegedly means “hardware having distinct structural characteristics that process bit lengths differently depending on required precision”. Remarks at 10-11. However, the definition of “type” proffered by Applicant is not given in the instant specification, nor, to Examiner’s knowledge, is that the commonly accepted definition in the art. Therefore, to the extent that each of the MACs has any distinguishing factor, including physical location, each MAC may be considered a different “type” under the broadest reasonable interpretation of that term in light of the specification. Here, since each MAU of Brothers is present in a different location, each one is of a different type. Applicant’s further contention that Brothers fails to disclose mapping feature groups based on a pre-generated mapping table, that argument is moot in light of the use of Lee to teach that limitation.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN C VAUGHN whose telephone number is (571)272-4849. The examiner can normally be reached M-R 7:00a-5:00p ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar, can be reached at 571-272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN C VAUGHN/Primary Examiner, Art Unit 2125
Prosecution Insights