DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/21/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: logic first recited in claim 1
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
Logic (first recited in claim 1): see Fig. 2 reference numeral 62 and 66; Fig. 10 reference numeral 354 and paragraph [0047] the logic includes a convolution streamer and a shared MAC hardware
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-17, 19-20 and 22-25 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “the logic is to swap weight inputs to the shared MAC hardware with activation inputs to the shared MAC hardware based on convolution type” in lines 10-12. A definition of swap is to exchange one thing for another. If the weight inputs are swapped with the activation input, it is unclear whether streaming the plurality of convolution operations requires streaming the activation inputs twice and not streaming the weight inputs since the weight inputs are swapped with the activation inputs. Further, clarification is required. For purposes of examination, this is interpreted as swapping data paths of the weight inputs with activation inputs to the shared MAC hardware and vice versa based on convolution type. Claim 9 recites a similar limitation and is rejected for the same reason. Claims 2-8 inherit the same deficiency as claim 1 by reason of dependence. Claims 10-17 inherit the same deficiency as claim 9 by reason of dependence.
Claim 3 recites “selectively enable multipliers of the adder tree structure during the one or more 2D convolution operations based on filter size”. It is unclear how there are multipliers in the adder tree structure since an adder tree is an arrangement of parallel adders in a hierarchical, tree-like structure. For purposes of examination, this is interpreted as selectively enable multipliers of the MAC hardware during the one or more 2D convolution operations based on filter size. Claims 11, 20 and 24 recite a similar limitation and are rejected for the same reason.
Claim 4 recites “wherein a utilization of the MAC hardware during the one or more 2D convolution operations is to be a function of a filter size”. It is unclear whether the utilization is a current function or a future function of the filter size because of the phrase “to be”. For purposes of examination, this is interpreted as wherein a utilization of the MAC hardware during the one or more 2D convolution operations is a function of a filter size. Claims 12 recites a similar limitation and are rejected for the same reason.
Claim 5 recites “wherein a utilization of the MAC hardware during the one or more 1D operations is to be a full utilization”. It is unclear whether the utilization is a current function or a future function of the filter size because of the phrase “to be”. For purposes of examination, this is interpreted as wherein a utilization of the MAC hardware during the one or more 1D operations is a full utilization. Claims 13 recites a similar limitation and are rejected for the same reason.
Claim 19 recites “wherein a number of cycles in the multi-cycle multiplication operation is to be a function of filter size”. It is unclear whether the number of cycles is a current function or a future function of the filter size because of the phrase “to be”. For purposes of examination, this is interpreted as wherein a number of cycles in the multi-cycle multiplication operation is a function of filter size. Claims 23 recites a similar limitation and is rejected for the same reason.
Claim 22 recites “the local memory” in lines 11-12. There is insufficient antecedent basis for this limitation in the claim. For purposes of examination, this is interpreted as a local memory. Claims 23-25 inherit the same deficiency as claim 24 by reason of dependence.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 18-25 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al. (US 20220335282 A1), hereinafter Park, in view of Nagamatsu (US 20220300253 A1) and Riegler et al. (US 20210012576 A1), hereinafter Riegler.
Regarding claim 18, Park teaches a computing system comprising:
a processor (Park Fig. 4 and paragraph [0131] processor – NPU; local memory – internal memory; logic – controller and PE array):
(Park paragraphs [0182, 0194, 0204, 0298-0300] plurality of convolution operations – point-wise and depth-wise convolution operations; 1D convolution operations - point-wise convolution operations; 2D convolution operations - depth-wise convolution operations),
stream the plurality of convolution operations to shared multiply-accumulate (MAC) hardware of the logic (Park Figs. 7-12 and paragraphs [0141-0143, 0186-0187, 0308] shared multiply-accumulate (MAC) hardware – PE array), and
store output data associated with the plurality of convolution operations to the local memory (Park paragraphs [0266, 0296] “The operation result data (i.e., output feature map data) calculated through the operation may be output from each PE and stored in the internal memory 200 or the feature map storage unit 220”; output data - operation result data).
Park does not explicitly teach a network controller; and a processor coupled the network controller, wherein the processor includes a local memory and logic coupled to one or more substrates, the logic to chain a plurality of convolution operations together.
However, on the same field of endeavor, Nagamatsu discloses chaining a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations, and wherein each of the one or more 2D convolution operations includes a multi-cycle multiplication operation (Nagamatsu Figs. 3-5 and 8-13 and paragraph [0051] “Depthwise Convolution is performed on the input feature map 21 to generate intermediate data 26. Then, as illustrated in "b" in the drawing, pointwise convolution, which is a 1x1 convolution operation, is performed on the generated intermediate data 26 using the pointwise convolution kernel 28, and an output feature map 29 is generated”; paragraphs [0096-0097]).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Park using Nagamatsu and chain together the depth-wise convolution operations and the point-wise convolution operations in order to implement a depthwise, pointwise separable convolution operation in a convolution layer of a convolution neural network (CNN) to reduce the amount of operation and the number of parameters in the convolution layer (Nagamatsu paragraphs [0003, 0007, 0050]).
Therefore, the combination of Park as modified in view of Nagamatsu teaches chain a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations, and wherein each of the one or more 2D convolution operations includes a multi-cycle multiplication operation.
Park as modified in view of Nagamatsu does not explicitly teach a network controller; and a processor coupled the network controller, wherein the processor includes a local memory and logic coupled to one or more substrates.
However, on the same field of endeavor, Riegler discloses a network controller; and a processor coupled the network controller, and logic coupled to one or more substrates (Riegler Figs. 7-8 and paragraphs [0040-0044, 0060] network controller – 126; processor – 112, 120 or 121; one or more substrates – 142; logic -144).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Park using Riegler and configure the system to include a network controller coupled the processor, and one or more substrates coupled to the logic in order to implement the system as a semiconductor apparatus (Riegler paragraph [0043]; Park paragraph [0102]).
Therefore, the combination of Park as modified in view of Nagamatsu and Riegler teaches a network controller; and a processor coupled the network controller, wherein the processor includes a local memory and logic coupled to one or more substrates.
Regarding claim 19, Park as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 18 as stated above. Further, Park as modified in view of Nagamatsu and Riegler teaches wherein a number of cycles in the multi-cycle multiplication operation is to be a function of filter size (Park paragraphs [0239, 0291-0292, 0298-0300]).
Regarding claim 20, Park as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 18 as stated above. Further, Park as modified in view of Nagamatsu and Riegler teaches wherein one or more of an adder tree structure or an accumulator of the MAC hardware is fixed between the one or more 1D convolution operations and the one or more 2D convolution operations, and wherein the logic is further to selectively enable multipliers of the adder tree structure during the one or more 2D convolution operations based on filter size (Park Fig. 10 and paragraphs [0307-0310]).
Regarding claim 21, Park as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 18 as stated above. Further, Park as modified in view of Nagamatsu and Riegler teaches wherein the one or more 1D convolution operations include pixel wise convolution operations and the one or more 2D convolution operations include depth wise convolution operations (Park paragraph [0235] “the first mode may mean an operation mode for … a point-wise convolution operation, and the second mode may mean an operation mode for a depth-wise convolution operation”; pixel wise convolution operations - point-wise convolution operation; depth wise convolution operations - depth-wise convolution operation; paragraph [0194]).
Regarding claim 22, Park teaches a semiconductor apparatus comprising:
logic ( Park Fig. 4 and paragraph [0131] logic – NPU):
(Park paragraphs [0182, 0194, 0204, 0298-0300] plurality of convolution operations – point-wise and depth-wise convolution operations; 1D convolution operations - point-wise convolution operations; 2D convolution operations - depth-wise convolution operations),
stream the plurality of convolution operations to shared multiply-accumulate (MAC) hardware of the logic (Park Figs. 7-12 and paragraphs [0141-0143, 0186-0187, 0308] shared multiply-accumulate (MAC) hardware – PE array), and
store output data associated with the plurality of convolution operations to the local memory (Park paragraphs [0266, 0296] “The operation result data (i.e., output feature map data) calculated through the operation may be output from each PE and stored in the internal memory 200 or the feature map storage unit 220”; output data - operation result data; local memory – internal memory).
Park does not explicitly teach one or more substrates; and logic coupled to the one or more substrates, the logic to: chain a plurality of convolution operations together.
However, on the same field of endeavor, Nagamatsu discloses chaining a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations, and wherein each of the one or more 2D convolution operations includes a multi-cycle multiplication operation (Nagamatsu Figs. 3-5 and 8-13 and paragraph [0051] “Depthwise Convolution is performed on the input feature map 21 to generate intermediate data 26. Then, as illustrated in "b" in the drawing, pointwise convolution, which is a 1x1 convolution operation, is performed on the generated intermediate data 26 using the pointwise convolution kernel 28, and an output feature map 29 is generated”; paragraphs [0096-0097]).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Park using Nagamatsu and chain together the depth-wise convolution operations and the point-wise convolution operations in order to implement a depthwise, pointwise separable convolution operation in a convolution layer of a convolution neural network (CNN) to reduce the amount of operation and the number of parameters in the convolution layer (Nagamatsu paragraphs [0003, 0007, 0050]).
Therefore, the combination of Park as modified in view of Nagamatsu teaches chain a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations, and wherein each of the one or more 2D convolution operations includes a multi-cycle multiplication operation.
Park as modified in view of Nagamatsu does not explicitly teach one or more substrates; and logic coupled to the one or more substrates.
However, on the same field of endeavor, Riegler discloses one or more substrates; and logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable or fixed-functionality hardware (Riegler Fig. 8 and paragraph [0043] one or more substrates – 142; logic – 144).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Park using Riegler and configure the system to include one or more substrates coupled to the logic in order to implement the system as a semiconductor apparatus (Riegler paragraph [0043]; Park paragraph [0102]).
Therefore, the combination of Park as modified in view of Nagamatsu and Riegler teaches one or more substrates; and logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable or fixed-functionality hardware.
Regarding claim 23, Park as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 22 as stated above. Further, Park as modified in view of Nagamatsu and Riegler teaches wherein a number of cycles in the multi-cycle multiplication operation is to be a function of filter size (Park paragraphs [0239, 0291-0292, 0298-0300]).
Regarding claim 24, Park as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 22 as stated above. Further, Park as modified in view of Nagamatsu and Riegler teaches wherein one or more of an adder tree structure or an accumulator of the MAC hardware is fixed between the one or more 1D convolution operations and the one or more 2D convolution operations, and wherein the logic is further to selectively enable multipliers of the adder tree structure during the one or more 2D convolution operations based on filter size (Park Fig. 10 and paragraphs [0307-0310]).
Regarding claim 25, Park as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 22 as stated above. Further, Park as modified in view of Nagamatsu and Riegler teaches wherein the one or more 1D convolution operations include pixel wise convolution operations and the one or more 2D convolution operations include depth wise convolution operations (Park paragraph [0235] “the first mode may mean an operation mode for … a point-wise convolution operation, and the second mode may mean an operation mode for a depth-wise convolution operation”; pixel wise convolution operations - point-wise convolution operation; depth wise convolution operations - depth-wise convolution operation; paragraph [0194]).
Claims 1-2, 6-10 and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Tsai et al. (US 20220083500 A1), hereinafter Tsai, in view of Nagamatsu and Riegler.
Regarding claim 1, Tsai teaches a computing system comprising:
a processor (Tsai Fig. 2 processor – 200; local memory – buffers; logic – 203 and 204):
(Tsai paragraph [0045] 1D convolution operations - 1D CNN algorithm; 2D convolution operations - 2D CNN algorithm),
stream the plurality of convolution operations to shared multiply-accumulate (MAC) hardware of the logic, wherein to stream the plurality of convolution operations to the shared MAC hardware, the logic is to swap weight inputs to the shared MAC hardware with activation inputs to the shared MAC hardware based on convolution type (Tsai Figs. 7A and 7C and paragraphs [0103 and 0107] “Each PE uses the multicast buffer to store a row of the input activation vectors, including the input halos, and uses the unicast buffer to store vectors of weights, as shown in FIG. 7A. For a 1D CONV with a filter width of 3, each flexible tensor accelerator PE will make three passes through the input activation buffer, exploiting the ID sliding window reuse … Irregular CONV kernels like depth-wise CONV have much less data reuse than regular CONV. Therefore, to support these workloads, the flexible tensor accelerator swaps the buffer usage in each ID CONV PE, as shown in FIG. 7C. Weights use the multicast buffer, while input activations use the unicast buffer”; multiply-accumulate (MAC) hardware – datapath 204), and
store output data associated with the plurality of convolution operations to the local memory (Tsai Fig. 2 and paragraph [0062] “The PE 202 is the core computation element that buffers the inputs, uses a datapath 204 to perform the tensor operation, and stores the result in an accumulator buffer”; output data – result).
Tsai does not explicitly teach a network controller; and a processor coupled the network controller, wherein the processor includes a local memory and logic coupled to one or more substrates, the logic to: chain a plurality of convolution operations together.
However, on the same field of endeavor, Nagamatsu discloses chaining a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations (Nagamatsu Figs. 3-5 and 8-13 and paragraph [0051] “Depthwise Convolution is performed on the input feature map 21 to generate intermediate data 26. Then, as illustrated in "b" in the drawing, pointwise convolution, which is a 1x1 convolution operation, is performed on the generated intermediate data 26 using the pointwise convolution kernel 28, and an output feature map 29 is generated”).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Tsai using Nagamatsu and chain together a plurality of convolution operations including depth-wise convolution operations and point-wise convolution operations in order to implement a depthwise, pointwise separable convolution operation in a convolution layer of a convolution neural network (CNN) to reduce the amount of operation and the number of parameters in the convolution layer (Nagamatsu paragraphs [0003, 0007, 0050]).
Therefore, the combination of Tsai as modified in view of Nagamatsu teaches chain a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations.
Tsai as modified in view of Nagamatsu does not explicitly teach a network controller; and a processor coupled the network controller, wherein the processor includes a local memory and logic coupled to one or more substrates.
However, on the same field of endeavor, Riegler discloses a network controller; and a processor coupled the network controller, and logic coupled to one or more substrates (Riegler Figs. 7-8 and paragraphs [0040-0044, 0060] network controller – 126; processor – 112, 120 or 121; one or more substrates – 142; logic -144).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Tsai using Riegler and configure the system to include a network controller coupled the processor, and one or more substrates coupled to the logic in order to implement the system as a semiconductor apparatus (Riegler paragraph [0043]).
Therefore, the combination of Tsai as modified in view of Nagamatsu and Riegler teaches a network controller; and a processor coupled the network controller, wherein the processor includes a local memory and logic coupled to one or more substrates.
Regarding claim 2, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 1 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein one or more of an adder tree structure or an accumulator of the MAC hardware is fixed between the one or more 1D convolution operations and the one or more 2D convolution operations (Tsai Figs. 4A, 7A and 7C and paragraphs [0103, 0107] adder tree structure – adder tree).
Regarding claim 6, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 1 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein the plurality of convolution operations further include one or more three-dimensional (3D) convolution operations (Tsai paragraph [0023] 3D convolution operations – 3D CNN algorithm).
Regarding claim 7, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 1 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein the one or more 1D convolution operations include pixel wise convolution operations and the one or more 2D convolution operations include depth wise convolution operations (Nagamatsu Fig. 3 and paragraphs [0051-0053] pixel wise convolution operations – point-wise convolution operations; depth wise convolution operations – depth-wise convolution operations).
Regarding claim 8, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 1 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein to stream the plurality of convolution operations to the shared MAC hardware, the logic is further to adjust convolution parameters to the shared MAC hardware based on the weight inputs (Tsai Figs. 1A and 1B and paragraphs [0024, 0046] convolution parameters – data movement).
Regarding claim 9, Tsai teaches a semiconductor apparatus comprising:
logic (Tsai Fig. 2 logic – 200):
(Tsai paragraph [0045] 1D convolution operations - 1D CNN algorithm; 2D convolution operations - 2D CNN algorithm);
stream the plurality of convolution operations to shared multiply-accumulate (MAC) hardware of the logic, wherein to stream the plurality of convolution operations to the shared MAC hardware, the logic is to swap weight inputs to the shared MAC hardware with activation inputs to the shared MAC hardware based on convolution type (Tsai Figs. 7A and 7C and paragraphs [0103 and 0107] “Each PE uses the multicast buffer to store a row of the input activation vectors, including the input halos, and uses the unicast buffer to store vectors of weights, as shown in FIG. 7A. For a 1D CONV with a filter width of 3, each flexible tensor accelerator PE will make three passes through the input activation buffer, exploiting the ID sliding window reuse … Irregular CONV kernels like depth-wise CONV have much less data reuse than regular CONV. Therefore, to support these workloads, the flexible tensor accelerator swaps the buffer usage in each ID CONV PE, as shown in FIG. 7C. Weights use the multicast buffer, while input activations use the unicast buffer”; multiply-accumulate (MAC) hardware – datapath 204); and
store output data associated with the plurality of convolution operations to a local memory (Tsai Fig. 2 and paragraph [0062] “The PE 202 is the core computation element that buffers the inputs, uses a datapath 204 to perform the tensor operation, and stores the result in an accumulator buffer”; output data – result).
Tsai does not explicitly teach one or more substrates; and logic coupled to the one or more substrates, the logic to: chain a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations.
However, on the same field of endeavor, Nagamatsu discloses chaining a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations (Nagamatsu Figs. 3-5 and 8-13 and paragraph [0051] “Depthwise Convolution is performed on the input feature map 21 to generate intermediate data 26. Then, as illustrated in "b" in the drawing, pointwise convolution, which is a 1x1 convolution operation, is performed on the generated intermediate data 26 using the pointwise convolution kernel 28, and an output feature map 29 is generated”).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Tsai using Nagamatsu and chain together a plurality of convolution operations including depth-wise convolution operations and point-wise convolution operations in order to implement a depthwise, pointwise separable convolution operation in a convolution layer of a convolution neural network (CNN) to reduce the amount of operation and the number of parameters in the convolution layer (Nagamatsu paragraphs [0003, 0007, 0050]).
Therefore, the combination of Tsai as modified in view of Nagamatsu teaches chain a plurality of convolution operations together, wherein the plurality of convolution operations include one or more one-dimensional (1D) convolution operations and one or more two-dimensional (2D) convolution operations.
Tsai as modified in view of Nagamatsu does not explicitly teach one or more substrates; and logic coupled to the one or more substrates.
However, on the same field of endeavor, Riegler discloses one or more substrates; and logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable or fixed-functionality hardware (Riegler Fig. 8 and paragraph [0043] one or more substrates – 142; logic – 144).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Park using Riegler and configure the system to include one or more substrates coupled to the logic in order to implement the system as a semiconductor apparatus (Riegler paragraph [0043]; Park paragraph [0102]).
Therefore, the combination of Park as modified in view of Nagamatsu and Riegler teaches one or more substrates; and logic coupled to the one or more substrates, wherein the logic is implemented at least partly in one or more of configurable or fixed-functionality hardware.
Regarding claim 10, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 9 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein one or more of an adder tree structure or an accumulator of the MAC hardware is fixed between the one or more 1D convolution operations and the one or more 2D convolution operations (Tsai Figs. 4A, 7A and 7C and paragraphs [0103, 0107] adder tree structure – adder tree).
Regarding claim 14, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 9 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein the plurality of convolution operations further include one or more three-dimensional (3D) convolution operations (Tsai paragraph [0023] 3D convolution operations – 3D CNN algorithm).
Regarding claim 15, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 9 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein the one or more 1D convolution operations include pixel wise convolution operations and the one or more 2D convolution operations include depth wise convolution operations (Nagamatsu Fig. 3 and paragraphs [0051-0053] pixel wise convolution operations – point-wise convolution operations; depth wise convolution operations – depth-wise convolution operations).
Regarding claim 16, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 9 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein to stream the plurality of convolution operations to the shared MAC hardware, the logic is further to adjust convolution parameters to the shared MAC hardware based on the weight inputs (Tsai Figs. 1A and 1B and paragraphs [0024, 0046] convolution parameters – data movement).
Regarding claim 17, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 9 as stated above. Further, Tsai as modified in view of Nagamatsu and Riegler teaches wherein the logic coupled to the one or more substrates includes transistor channel regions that are positioned within the one or more substrates (Riegler paragraph [0044]). The motivation to combine is the same as claim 9.
Claims 3-5 and 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Tsai in view of Nagamatsu and Riegler as applied to claims 1-2 and 9-10 above, and further in view of Park.
Regarding claim 3, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 2 as stated above.
Tsai does not explicitly teach wherein the logic is further to selectively enable multipliers of the adder tree structure during the one or more 2D convolution operations based on filter size.
However, on the same field of endeavor, Park discloses selectively enabling multipliers or processing elements of a processing element array during one or more 2D convolution operations based on filter size (Park Fig. 10 and paragraphs [0298-0300, 0307-0310]).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Tsai using Park and configure the logic to selectively enable multipliers of the adder tree structure during the one or more 2D convolution operations based on filter size in order to deactivate components not in used to minimize power consumption of the system (Park paragraphs [0310-0312]).
Therefore, the combination of Park as modified in view of Nagamatsu, Riegler and Park teaches wherein the logic is further to selectively enable multipliers of the adder tree structure during the one or more 2D convolution operations based on filter size.
Regarding claim 4, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 1 as stated above.
Tsai does not explicitly teach wherein a utilization of the MAC hardware during the one or more 2D convolution operations is to be a function of a filter size.
However, on the same field of endeavor, Park discloses a utilization of a MAC hardware during one or more 2D convolution operations is a function of a filter size (Park Figs. 9-10 and paragraphs [0298-0301, 0307-0310]).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Tsai using Park and configure the utilization of the MAC hardware during the one or more 2D convolution operations as a function of the filter size such that the utilization rate of the MAC hardware is increased during the depth-wise convolution operation of the MAC hardware and to deactivate components that are not being used minimize power consumption of the system (Park paragraphs [0310-0312]).
Therefore, the combination of Park as modified in view of Nagamatsu, Riegler and Park teaches wherein a utilization of the MAC hardware during the one or more 2D convolution operations is to be a function of a filter size.
Regarding claim 5, Tsai as modified in view of Nagamatsu and Riegler teaches all the limitations of claim 1 as stated above.
Tsai does not explicitly teach wherein a utilization of the MAC hardware during the one or more 1D operations is to be a full utilization.
However, on the same field of endeavor, Park discloses a utilization of a MAC hardware during one or more 1D operations is a full utilization (Park Figs. 8 and 10 and paragraphs [0308-0310]).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, to modify Tsai using Park and configure the utilization of the MAC hardware during the one or more 1D operations as a full utilization in order to maximize the utilization rate of the MAC hardware when performing point-wise convolution operations (Park paragraph [0021, 0023, 0195, 0307-0310]).
Therefore, the combination of Park as modified in view of Nagamatsu, Riegler and Park teaches wherein a utilization of the MAC hardware during the one or more 1D operations is to be a full utilization.
Regarding claims 11-13, they recite substantially the same limitations as claims 3-5 respectively. Claims 3-5analysis applies equally to claims 11-13 respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lee et al. (US 20200202198 A1) generally related to a utilization rate of a MAC hardware based on filter size when performing 2D convolutions.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Carlo Waje whose telephone number is (571)272-5767. The examiner can normally be reached 9:00-6:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James Trujillo can be reached at (571) 272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Carlo Waje/Examiner, Art Unit 2151 (571)272-5767
Prosecution Insights