DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 1 objected to because of the following informalities: Claim 1 recites “a first deep learning,” instead of a ‘first deep learning operation’ as is recited later in the claims. Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Jianchao Yang et al. (“BP-Im2col: Implicit Im2col Supporting AI Backpropagation” 2022).
Regarding claim 1, Yang teaches;
A method for deep learning operations, the method comprising: storing one or more data elements in a tensor ([pg. 5] Transposed convolution mode … four-dimensional convoluted feature map … which is actually stored in the on-chip buffer) of a first deep learning [operation] ([pg. 2] loss calculation is realized by performing transposed convolution on the loss of the output by the convolving kernel) in a memory ([pg. 5] stored in the on-chip buffer)
generating a bitmap ([pg. 6] We detect each address according to Section 4.2 to filter the zero pixels, and perform address mapping to generate … the mask) based on one or more parameters of the first deep learning operation ([pg. 2] 𝐾ℎ, 𝐾𝑤 ... height and width of 𝑊l (kernel) … 𝑆, 𝑃ℎ, 𝑃𝑤 Stride, padding size in height and width directions [pg. 5, section 4.2] Transposed convolution mode … if ℎ < 𝐾ℎ − 1 − 𝑃ℎ or 𝑤 < 𝐾𝑤 – 1 – 𝑃𝑤 or [ℎ − (𝐾ℎ − 1 − 𝑃ℎ)]%𝑆 > 0 or [𝑤 − (𝐾𝑤 – 1 − 𝑃𝑤)]%𝑆 > 0 then 𝑎𝑑𝑑𝑟𝑜𝑢𝑡 = 𝑁𝑈𝐿𝐿; //Zero-spaces)
NOTE: Yang determines the zero-space pattern from the transposed-convolution parameters (kernel size, stride, padding size), and generates a mask reflecting those determinations.
the bitmap comprising bits indicating whether data elements in an upsampled tensor ([pg. 5] four-dimensional convoluted feature map with zero-spaces) are in the tensor, ([pg. 5, section 4.2] Output: Address in the original feature map without zero-spaces, … 𝑎𝑑𝑑𝑟𝑜𝑢𝑡 = 𝑁𝑈𝐿𝐿; // Zero-spaces [pg. 6] We detect each address according to Section 4.2 to filter the zero pixels, and perform address mapping to generate the compressed address of the actually stored feature map and the mask to send to buffer B)
PNG
media_image1.png
936
1090
media_image1.png
Greyscale
NOTE: Fig. 5 depicts Yang’s mask as a plurality of bits and further illustrates that mask positions having a value of 0 correspond to invalid addresses, while a position having a value of 1 corresponds to a valid address for data stored in buffer B. This is further supported in Algorithm 1, which returns 𝑎𝑑𝑑𝑟𝑜𝑢𝑡 = 𝑁𝑈𝐿𝐿 for zero spaces. Thus, the mask bits indicate whether respective positions of the virtual upsampled tensor correspond to data elements of the original stored tensor or to virtual zero-space positions.
the upsampled tensor ([pg. 5] four-dimensional convoluted feature map with zero-spaces) comprising one or more data elements than the tensor ([pg. 4] four-dimensional tensor … 𝐵×𝐶×𝐻×𝑊. Suppose the numbers of zero-paddings of this tensor are 𝑧𝑃ℎ and 𝑧𝑃𝑤 …, and the number of zero-insertions is 𝑧𝑆−1 … The dimension of the tensor after reorganization is 𝐻′′ =𝐻+2×𝑧𝑃ℎ+(𝐻−1)×(𝑧𝑆−1), 𝑊′′ =𝑊+2×𝑧𝑃𝑤+(𝑊−1)×(𝑧𝑆−1));
generating one or more storage pointers indicating one or more memory addresses of the one or more data elements of the tensor in the memory ([pg. 4-5] original four-dimensional feature map, which is actually stored in memory [pg. 5, Section 4.2, Algorithm 1] Output: Address in the original feature map without zero-spaces, 𝑎𝑑𝑑𝑟𝑜𝑢𝑡); retrieving the one or more data elements from the memory based on the one or more storage pointers ([pg. 6] We detect each address according to Section 4.2 and perform address mapping to generate the compressed address of the actually stored feature map and the mask to send to buffer 𝐵. After the data is transmitted back, we send it directly to the PEs);
NOTE: Yang’s generated 𝑎𝑑𝑑𝑟𝑜𝑢𝑡 constitutes a storage pointer because it identifies an address in the original feature map stored in the on-chip buffer.
and performing one or more second deep learning operations ([pg. 2] 𝛿𝐼𝑙+1𝑒𝑖 ∗ … ∗ The convolution symbol) on the upsampled tensor ([pg. 2] 𝛿𝐼𝑙+1𝑒𝑖 … 𝑒/𝑖/𝑒𝑖 Zero-paddings, zero-insertions, and both) using the bitmap and the one or more data elements ([pg. 6] After the data is transmitted back, we send it directly to the PEs, according to its compressed mask. When the data enters the systolic array, the zero pixel position identified by the compressed mask is temporarily filled with zeros) to compute one or more outputs of the first deep learning operation ([pg. 2] Equation 1 … 𝛿𝐼𝑙 = 𝛿𝐼𝑙+1𝑒𝑖 ∗ 𝑇𝑟(𝑟𝑜𝑡180◦𝑊𝑙) … loss calculation is realized by performing transposed convolution on the loss of the output by the convolving kernel. The convolving kernel is rotated and transposed, and data reorganization of zero-insertions and zero-paddings is subsequently performed for the loss of the output (see Equation 1)).
Regarding claim 2, Yang teaches;
wherein the first deep learning operation is a transposed convolution or a dilated convolution ([pg. 2] Equation 1 … 𝛿𝐼𝑙 =𝛿𝐼𝑙+1𝑒𝑖 ∗ 𝑇𝑟(𝑟𝑜𝑡180◦𝑊𝑙) … loss calculation is realized by performing transposed convolution on the loss of the output by the convolving kernel), and a second deep learning operation is a convolution ([pg. 2] 𝛿𝐼𝑙+1𝑒𝑖 ∗ … ∗ The convolution symbol).
Regarding claim 3, Yang teaches;
wherein the tensor ([pg. 2] 𝛿𝐼𝑙+1) is an input tensor of the convolution (NOTE: Yang’s 𝛿𝐼𝑙+1 provides the original stored input data corresponding to non-zero space positions for the convolution, while 𝛿𝐼𝑙+1𝑒𝑖 is the corresponding zero-inserted/padded representation [pg. 2] 𝛿𝐼𝑙+1𝑒𝑖 ∗ … ∗ The convolution symbol … [pg. 5] We map the addresses of virtual matrix 𝐵 to the virtual four-dimensional convoluted feature map with zero-spaces, then map it to the four-dimensional convoluted feature map without zero-spaces, which is actually stored in the on-chip buffer), the one or more outputs ([pg. 2] 𝛿𝐼𝑙) are in an output tensor of the convolution ([pg. 2] 𝛿𝐼𝑙 =𝛿𝐼𝑙+1𝑒𝑖 ∗ 𝑇𝑟(𝑟𝑜𝑡180◦𝑊𝑙) … ∗ The convolution symbol),
(NOTE: 𝐻𝑜′′′ > 𝐻𝑖 > 𝐻𝑜, see reasoning below this limitation) and a dimension of the output tensor ([pg. 2] 𝐻𝑖 … height … of 𝐼𝑙) of the convolution is smaller than a dimension of the upsampled tensor ([pg. 2] 𝐻𝑜′′′) but larger than a dimension of the input tensor ([pg. 2] 𝐻𝑜, … Height … of … 𝐼𝑙+1).
Reasoning:
Claim: upsampled tensor dimension (𝐻𝑜′′′) > output tensor dimension (𝐻i) > input tensor dimension (𝐻𝑜)
([pg. 2]
𝐻𝑖, ... height ... of 𝐼𝑙 // output tensor height
𝐻𝑜, ... Height ... of ... 𝐼𝑙+1 // input tensor height
𝐻𝑜′′′ = 𝐻𝑜 + 2 · (𝐾ℎ−1−𝑃ℎ) + (𝐻𝑜−1) × (𝑆−1) // upsampled tensor height
stride of the transposed convolution is a fixed value of 1)
// Thus, when computing 𝐻i, the stride is 1.
([pg. 3]
𝑝𝑎𝑑𝑑𝑖𝑛𝑔 = 1 and 𝑠𝑡𝑟𝑖𝑑𝑒 = 2)
// Stride of the original convolution is 2 (Thus, when computing 𝐻𝑜′′′, S is 2)
// padding = 1 (Ph = 1)
// Yang depicts 3 x 3 kernel in fig. 2 (Kh = 3)
// Thus:
// 𝐻𝑜′′′ = 𝐻𝑜 + 2 · (3-1-1) + (𝐻𝑜-1) × (2-1)
// = 𝐻𝑜 + 2 + 𝐻𝑜 - 1
// = 2𝐻𝑜 + 1
// convolving a 2𝐻𝑜 + 1 high tensor 𝐻𝑜′′′ with a 3 high kernel and stride 1 produces:
// 𝐻i = (𝐻𝑜′′′ - Kh)/1 + 1
= ((2𝐻𝑜 + 1) – 3)/1 + 1
= 2𝐻𝑜 – 1
// Therefore, for 𝐻𝑜 > 1: upsampled tensor dimension (𝐻𝑜′′′ = 2Ho + 1) > output tensor dimension (𝐻i = 2𝐻𝑜 - 1) > input tensor dimension (𝐻𝑜)
Regarding claim 4, Yang teaches;
determining one or more positions where one or more additional elements ([pg. 5] Zero-spaces) are inserted into the tensor based on the one or more parameters ([pg. 2] 𝐾ℎ, 𝐾𝑤 ... height and width of 𝑊l (kernel) … 𝑆, 𝑃ℎ, 𝑃𝑤 Stride, padding size in height and width directions [pg. 5, section 4.2] Transposed convolution mode … if ℎ < 𝐾ℎ − 1 − 𝑃ℎ or 𝑤 < 𝐾𝑤 – 1 – 𝑃𝑤 or [ℎ − (𝐾ℎ − 1 − 𝑃ℎ)]%𝑆 > 0 or [𝑤 − (𝐾𝑤 – 1 − 𝑃𝑤)]%𝑆 > 0 then 𝑎𝑑𝑑𝑟𝑜𝑢𝑡 = 𝑁𝑈𝐿𝐿; //Zero-spaces); and generating the bitmap based on the one or more positions ([pg. 6] We detect each address according to Section 4.2 to filter the zero pixels, and perform address mapping to generate … the mask).
Regarding claim 5, Yang teaches;
the bitmap comprises one or more bits and one or more additional bits, the one or more bits have a first value and correspond to the one or more elements (NOTE: Yang’s mask comprises ‘1’ bits, corresponding to the stored elements of the original tensor. See fig. 5 below.), the one or more additional bits have a second value and correspond to the one or more additional elements (NOTE: Yang’s mask comprises ‘0’ bits, corresponding to positions of zero-spaces. See fig. 5 below.), and the first value (1) is different from the second value (0).
PNG
media_image1.png
936
1090
media_image1.png
Greyscale
Regarding claim 6, Yang teaches;
wherein the one or more parameters of the first deep learning operation comprises a padding size, a kernel size, a stride size, or a dilation rate ([pg. 2] 𝐾ℎ, 𝐾𝑤 ... height and width of 𝑊l (kernel) … 𝑆, 𝑃ℎ, 𝑃𝑤 Stride, padding size in height and width directions) of the first deep learning operation ([pg. 5] Transposed convolution mode … if ℎ < 𝐾ℎ − 1 − 𝑃ℎ or 𝑤 < 𝐾𝑤 – 1 – 𝑃𝑤 or [ℎ − (𝐾ℎ − 1 − 𝑃ℎ)]%𝑆 > 0 or [𝑤 − (𝐾𝑤 – 1 − 𝑃𝑤)]%𝑆 > 0 then 𝑎𝑑𝑑𝑟𝑜𝑢𝑡 = 𝑁𝑈𝐿𝐿; //Zero-spaces).
Regarding claim 7, Yang teaches;
wherein the one or more data elements are in a same channel ([pg. 5] Transposed convolution mode … for loss calculation … a certain pixel to be calculated is mapped to the position of one channel of the virtual feature map with zero-spaces as (ℎ,𝑤), … For the NZs (non-zeros) … the position in the virtual feature maps with zero-spaces is (ℎ,𝑤), while the target position in the actually stored feature maps is ([ℎ−(𝐾ℎ−1−𝑃ℎ)]/𝑆, [𝑤−(𝐾𝑤−1−𝑃𝑤)]/𝑆)) of the first deep learning operation ([pg. 2] loss calculation is realized by performing transposed convolution on the loss of the output by the convolving kernel).
Regarding claims 8-14,
Claims 8-14 are non-transitory computer-readable media claims directly corresponding to method claims 1-7 and are rejected using the same reasoning.
Regarding claims 15-20,
Claims 15-20 are apparatus claims directly corresponding to method claims 1-6 and are rejected using the same reasoning.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
