DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Remarks
Examiner notes that on Specification Paragraphs 71 and 87, the specification newly states the following instructions “a BacthNormStats” and “TransposeBacthNormStats”, this may be inconsistent with what the applicant wants, wherein paragraph 56 discloses the following instructions “BatchNormStats” and “TransposeBatchNormStats”.
Information Disclosure Statement
The information disclosure statements filed 02/22/2023, 05/10/2023, 08/11/2023, 10/27/2023, and 02/26/2026 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered.
Specification
The disclosure is objected to as failing to comply with 37 CFR 1.71(a) because of the following informalities:
Par.51, p.13, ll.4, “(column elements [64:32])” should read as “(column elements [63:32])”, in view of fig.5 (emphasis added).
Par.54, ll.1, “buffe” should read as “buffer”.
Par.60, ll.6, “elemengt” should read as “element”.
Par.62, ll.4, “TransposeBacthNormStats” should read as “TransposeBatchNormStats”, in view of par.56.
Par.90, ll.5, “1306 can be uses as a type of” should read as “1306 can be used as a type of” (emphasis added).
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 5, 7-9, 11-14, 16-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Abts et al. (US 12,340,300 B1), hereinafter Abts, and in view of Zejda et al. (US 11,036,827 B1), hereinafter Zejda.
Regarding claim 5, Abts discloses
A Vector Processing Tile [VXM] that has a plurality of Lanes (i.e. Compute Channels), each lane including a plurality of arithmetic logic unit (ALU) circuits coupled in series and wherein the plurality of lanes is operable to generate outputs in parallel [“VXM contains 16 [ALUs] per lane”, “supports chaining together two or more Vector ALUs within each lane”, and “efficient parallel implementations of algorithms for batch normalization…” see Col.12, The Vector Processing Tiles; “Each stream automatically progresses in its designated direction on every cycle” col.10-11, Streams];
And a transpose circuit, and wherein the transpose circuit is operable to receive an input tensor, transpose the input tensor, and output a transposed tensor [“The switch units [SXM”/”NET] execute functions for the transposition, permutation, shifting and rotation of data elements… these operations are used for performing tensor reshape operations common to machine learning algorithms… the SXM can rotate or transpose a stream of data across the lanes.” Col.13, The Switching Processing Tiles];
However, Abts does not explicitly disclose the plurality of compute channels coupled to the transpose circuit, and output a transposed tensor to the plurality of compute channels
In the analogous art of transposition circuitry architecture, Zejda teaches a transpose circuit coupled to the computing circuit, and wherein the transpose circuit is operable to receive an input tensor, transpose the input tensor, and output a transposed tensor to the computing circuit [Figure 7 and 9; “The buffer/transposer 702 may reformat the data from a host-friendly format (e.g., a row-major order) into a format used by the compute array 600 ( e.g., a column-major order)” col.10, ll.56-58; “The buffer/transposer may be suitable for use in applications other than GEMM” Col.13-14]
It would have been obvious to one of ordinary skill in the art, having the teachings of Abts and Zejda before him before the effective filing date of the claimed invention to modify the SXMs as taught by Abts, to include the transposition circuitry as input to the Vector Processing Tile as taught by Zejda, to ensure the proper format for the compute engines and reduce resource usage [Zejda: Col.10-11 and 13]. The combination of Abts and Zejda discloses the plurality of compute channels coupled to the transpose circuit, and output a transposed tensor to the plurality of compute channels
Regarding claim 7, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 5 above.
Abts discloses wherein the transpose circuit is operable to transpose the input tensor by receiving input elements corresponding to a column of the input tensor in parallel, and providing the input elements in series as a vector of the transposed tensor to a compute channel [“SXM can be used to arbitrarily remap the 16 lanes within each Superlane… A transpose operation takes 16 incoming streams and produces 16 output streams with the rows and columns exchanged” Col.13, The Switching Processing Tiles].
Zejda also discloses wherein the transpose circuit is operable to transpose the input tensor by receiving input elements corresponding to a column of the input tensor in parallel, and providing the input elements in series as a vector of the transposed tensor to a compute channel [See figures 7-9].
Regarding claim 8, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 5 above.
Abts discloses wherein the output of a compute channel is an output vector generated by applying an elementwise operation to each element of the vector of the tensor inputted into the compute channel [“The VXM also performs common normalization functions… supports chaining… vector ALUs… for batch normalization, quantization, or more complex activation functions” Col.12, The Vector Processing Tiles, discloses various element-wise operations; See Col.10-11, Stream, talks about vectors in the lanes].
However, Abts does not explicitly disclose applying an elementwise operation to each element of the vector of the transposed tensor inputted into the compute channel.
In the analogous art of transposition circuitry architecture, Zejda teaches a transpose circuit coupled to the computing circuit, and wherein the transpose circuit is operable to receive an input tensor, transpose the input tensor, and output a transposed tensor to the computing circuit [Figure 7 and 9; “The buffer/transposer 702 may reformat the data from a host-friendly format (e.g., a row-major order) into a format used by the compute array 600 ( e.g., a column-major order)” col.10, ll.56-58; “The buffer/transposer may be suitable for use in applications other than GEMM” Col.13-14]
It would have been obvious to one of ordinary skill in the art, having the teachings of Abts and Zejda before him before the effective filing date of the claimed invention to modify the SXMs as taught by Abts, to include the transposition circuitry as input to the Vector Processing Tile as taught by Zejda, to ensure the proper format for the compute engines and reduce resource usage [Zejda: Col.10-11 and 13]. The combination of Abts and Zejda discloses applying an elementwise operation to each element of the vector of the transposed tensor inputted into the compute channel.
Regarding claim 9, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 5 above.
Abts discloses wherein the output of a compute channel includes an output value generated by performing a computation on elements of the vector of the tensor inputted into the compute channel [“The VXM also performs common normalization functions… supports chaining… vector ALUs… for batch normalization, quantization, or more complex activation functions” Col.12, The Vector Processing Tiles, discloses various element-wise operations; See Col.10-11, Stream, talks about vectors in the lanes].
of the transposed tensor inputted into the compute channel.
However, Abts does not explicitly disclose performing a computation on elements of the vector of the transposed tensor inputted into the compute channel.
In the analogous art of transposition circuitry architecture, Zejda teaches a transpose circuit coupled to the computing circuit, and wherein the transpose circuit is operable to receive an input tensor, transpose the input tensor, and output a transposed tensor to the computing circuit [Figure 7 and 9; “The buffer/transposer 702 may reformat the data from a host-friendly format (e.g., a row-major order) into a format used by the compute array 600 ( e.g., a column-major order)” col.10, ll.56-58; “The buffer/transposer may be suitable for use in applications other than GEMM” Col.13-14]
It would have been obvious to one of ordinary skill in the art, having the teachings of Abts and Zejda before him before the effective filing date of the claimed invention to modify the SXMs as taught by Abts, to include the transposition circuitry as input to the Vector Processing Tile as taught by Zejda, to ensure the proper format for the compute engines and reduce resource usage [Zejda: Col.10-11 and 13]. The combination of Abts and Zejda discloses performing a computation on elements of the vector of the transposed tensor inputted into the compute channel.
Regarding claim 11, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 5 above.
Abts discloses wherein the plurality of compute channels and the transpose circuit are part of one of a plurality of vector compute banks [Superlanes] of the integrated circuit device [Fig.15B, Superlanes; “A Superlane processes streams of data in 16 lanes” Col.8-10, Superlanes].
Regarding claim 12, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 11 above.
Abts discloses wherein the plurality of vector compute banks is operable to perform transpose operations in parallel using the transpose circuit in each of the vector compute banks [“Each slice (for example, slice 1502) in a TSP performs any of a variety of functions under the control of instructions… All of the tiles in a slice execute the same set of instructions …each Superlane, and indeed the entire TSP, executes a single set of instructions” Col.8-9,ll.45-36; “The compiler coordinates the control and data flow of the program and specifies any instruction-level parallelism by explicitly bundling instructions that can and should execute concurrently so that they are dispatched together” Col.19-20, The Compiler; see fig.21 for concurrency and col.14-15].
Regarding claim 13, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 5 above.
Abts discloses wherein the transpose circuit includes a bypass mode of operation to output data to the plurality of compute channels without transposing the data. [“the results transferred from the VXM through the SXMs through the PCie modules back to the host computer” Col.10,ll.4-19, can merely be used to transfer data; “SXM can rotate or transpose a stream of data across the lanes.” Col. 13, Switching Processing Tiles, can perform other functions not only transposing; “All of the tiles in a slice execute the same set of instructions” Col.8,ll.60-65]
Zejda also discloses that if the data is in a compute-engine friendly format it does not need be transposed prior to input into the compute engine. See col.11, ll.13-20.
Regarding claims 14, 16-18 and 20, they are directed to claims 5, 7-9 and 13, respectively. A mere change in statutory class is obvious. The claims are rejected for the reasons given in the respective directed claim.
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Abts and Zejda, and further in view of Bleiweiss et al. (US 2019/0205737 A1), hereinafter Bleiweiss.
Regarding claim 6, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 5 above.
further comprising a processing engine array operable to perform matrix multiplication computations, wherein the transpose circuit is operable to perform transpose operations concurrently with matrix multiplication operations being performed by the processing engine array.
Abts discloses further comprising a processing engine array operable to perform matrix multiplication computations [Fig.18; wherein MXM can perform multiply-accumulate, and dot products, see Col.12-13 The Matrix Processing Tiles]
However, Abts and Zejda does not explicitly disclose wherein the transpose circuit is operable to perform transpose operations concurrently with matrix multiplication operations being performed by the processing engine array.
In the analogous art of neural network processing architecture, Bleiweiss discloses perform transpose operations concurrently with matrix multiplication operations being performed [“2D Convolution may perform better on a GPU” par.226; “Accelerator 2708 may be implemented to perform additional data deep layout transformations other than a transpose… data layout conversions may be offloaded to accelerator 2709 to enable parallel execution with other computations performed at the GPU” par.236]
It would have been obvious to one of ordinary skill in the art, having the teachings of Abts, Zejda, and Bleiweiss before him before the effective filing date of the claimed invention to modify the system as taught by Abts, to separate/offload layout transformations as taught by Bleiweiss, to enable parallel execution of various computations [Bleiweiss: par.234-238]. The combination of Abts, Zejda, and Bleiweiss discloses the transpose circuit is operable to perform transpose operations concurrently with matrix multiplication operations being performed by the processing engine array.
Regarding claim 15, it is directed to claim 6. A mere change in statutory class is obvious. The claim is rejected for the reasons given in the directed claim.
Claims 10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Abts and Zejda, and further in view of Bleiweiss , and further in view of Wu et al. (NPL: “Group Normalization”), hereinafter Wu.
Regarding claim 10, Abts and Zejda disclose the invention substantially as claimed. See the discussion of claim 9 above.
wherein the output value is a mean, a variance, or a count of the elements of the vector of the transposed tensor inputted into the compute channel.
Abts discloses performing batch normalizing or other complex functions using the compute channel [“The TSP supports chaining together two or more vector ALUs within each lane… this allows for efficient parallel implementations of algorithms for batch normalization, quantization, or more complex activation functions” Col.12, Vector Processing Tiles]
However, Abts does not explicitly disclose wherein the output value is a mean, a variance, or a count of the elements of the vector of the transposed tensor inputted into the compute channel.
In the analogous art of transposition circuitry architecture, Zejda teaches a transpose circuit coupled to the computing circuit, and wherein the transpose circuit is operable to receive an input tensor, transpose the input tensor, and output a transposed tensor to the computing circuit [Figure 7 and 9; “The buffer/transposer 702 may reformat the data from a host-friendly format (e.g., a row-major order) into a format used by the compute array 600 ( e.g., a column-major order)” col.10, ll.56-58; “The buffer/transposer may be suitable for use in applications other than GEMM” Col.13-14]
It would have been obvious to one of ordinary skill in the art, having the teachings of Abts and Zejda before him before the effective filing date of the claimed invention to modify the SXMs as taught by Abts, to include the transposition circuitry as input to the Vector Processing Tile as taught by Zejda, to ensure the proper format for the compute engines and reduce resource usage [Zejda: Col.10-11 and 13]. The combination of Abts and Zejda discloses the elements of the vector of the transposed tensor inputted into the compute channel.
However, Abts and Zejda does not explicitly disclose the output value is a mean, a variance, or a count of the elements of the vector of the tensor
In the analogous art of neural network processing architecture, Bleiweiss discloses computing the variables for normalization before computing the normalization [ “Batch Normalization involves normalization of an input by subtracting a mean and dividing the variance calculated over a mini-batch during training (and inference)” par.244, Fig.29 shows computing the mean and variance prior to the normalization operations; “Once calculated, the results of mean and variance operations 3003 are forwarded to complete computation of normalization operations 3004” par.245]
It would have been obvious to one of ordinary skill in the art, having the teachings of Abts, Zejda, and Bleiweiss before him before the effective filing date of the claimed invention to modify the Vector Processing Tile as disclosed by Abts, to compute the variance and mean for normalization as taught by Bleiweiss, to ensure to compute the inputs required for normalization, as normalization allows for larger learning rates, decreases sensitivity to initialization of weights, and in some cases aids in increase of accuracy [Bleiweiss: par.244-247]. The combination of Abts and Bleiweiss discloses the output value is a mean, or a variance of the elements of the vector of the tensor.
However, Abts, Zejda, and Bleiweiss does not explicitly disclose a count of the elements of the vector of the tensor.
In the analogous art of Neural Network Normalizations, Wu teaches that normalizations requires a count of the number of elements in order to compute the mean and variance [Fig.2; See Sec.3.1 “m is the size of this set. Many types of feature normalization methods main differ in how the set Si is defined” and equations 1-7].
It would have been obvious to one of ordinary skill in the art, having the teachings of Abts, Zejda, Bleiweiss, and Wu before him before the effective filing date of the claimed invention to modify the Vector Processing Tile as disclosed by Abts and Bleiweiss, to compute the variance, mean, and count for normalization as taught by Wu, to be able to compute the normalization of each element as it is well-known that normalizing the input data makes training faster [Wu: Sec.2 and 3.1].
Regarding claim 19, it is directed to claim 10. A mere change in statutory class is obvious. The claim is rejected for the reasons given in the directed claim.
Allowable Subject Matter
Claims 1-4 allowed.
The following is a statement of reasons for the indication of allowable subject matter:
Zejda et al. (US 11,003,429 B1), discloses the state buffer, processing engine array, results buffer, and various vector engines with a plurality of execution channels [Fig.6 and col.9], however does not explicitly disclose a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit, and wherein the transpose circuit is operable to offload transpose operations from the processing engine array.
Li et al. (US 2020/0409664 A1) discloses a state buffer memory having a plurality of row partitions organized into row groups, processing engine array, results buffer, various engines with a plurality of execution channels, and transposing with the processing engine array [fig.2-4, par.12-13, 31-35 and 41-45], however does not explicitly disclose a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit, and wherein the transpose circuit is operable to offload transpose operations from the processing engine array.
Weissenborn et al. (US 2021/0383199 a1) discloses the matrix can be transposed prior to normalization [fig.3, and par.66]. however does not explicitly disclose a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit, and wherein the transpose circuit is operable to offload transpose operations from the processing engine array.
Phelps et al. (US 2018/0336164 A1) discloses matrix multiply unit, vector processing unit, and transpose logic units [fig.1b-1c and 2] however does not explicitly disclose a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit.
Bradford et al. (US 2019/0042248 A1) discloses offloading the transpose operations onto a transpose engine [Fig.3B and par.69], however does not explicitly disclose a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit.
Liu et al. (US 2020/0311531 A1) discloses Vector computing unit circuit and matrix transposition/conversion circuit [fig.1a and 1d], however does not explicitly disclose a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit.
Yin et al. (US 2022/0284283 A1 from IDS filed 02/26/2026) discloses batch normalization, Matrix multiplication logic, various architectures, the use of transposition for various layers including normalization [fig.30, par.62, 79-86], however does not explicitly disclose a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit, and wherein the transpose circuit is operable to offload transpose operations from the processing engine array.
Kim et al. (US 2022/0283984 A1) discloses a neural network processor with MAC array, and normalization module for post processing and pre-processing for another convolution accelerator [fig.3].
Hargil et al. (US 2020/0356837 A1 from IDS filed 02/26/2026) discloses using a VPU for matrix multiplication, and transposition [Fig.4].
The prior art above alone or in combination with any other reference fails to disclose or provide motivation for the following limitation in combination with the rest of the claim limitations: “a vector compute engine having a plurality of vector compute banks coupled to respective row groups of the state buffer memory, each vector compute bank having a transpose circuit and a plurality of compute channels coupled to the transpose circuit, wherein the transpose circuit is operable to offload transpose operations from the processing engine array by transposing submatrices obtained from the row group corresponding to the vector compute bank, and the plurality of compute channels is operable to perform computations on transposed submatrices outputted from the transpose circuit.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ba et al. (NPL: “Layer Normalization”), discloses LayerNorm, and the transpose of batch normalization into layer normalization. See abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kenny K. Bui whose telephone number is (571)270-0604. The examiner can normally be reached 8:00 am to 3:00 pm on Monday, 8:00 am to 4:00 pm on Tuesday to Friday ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew T Caldwell can be reached at (571)272-3702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KENNY K. BUI/Patent Examiner, Art Unit 2182 (571)270-0604
/ANDREW CALDWELL/Supervisory Patent Examiner, Art Unit 2182
Prosecution Insights