Non-Final Office Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-13, 15, and 17-19 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mody et al., US 2020/0074287 A1.
Referring to claim 1:
In para. 0042 Mody et al. disclose an apparatus, comprising: matrix operation hardware circuitry (dot product computation).
In para. 0040 and 0042, Mody et al. disclose circuitry coupled to the matrix operation hardware circuitry to detect a hardware fault in the matrix operation hardware circuitry based at least in part on one or more hardware checksums of data in one or more matrices of the matrix operation hardware circuitry (checksums on the input matrices--0040 and compute expected product checksums of the output matrix C).
Referring to claim 2, in para. 0041, Mody et al. disclose wherein the circuitry is further to: correct the detected hardware fault based at least in part on the one or more hardware checksums (the difference between the computed checksum and the reference checksum can be used to determine the correct value).
Referring to claim 3, in para. 0037-0038, Mody et al. disclose safety critical operation defined by the Automotive Safety Integrity Levels (ASIL). In para. 0041, Mody et al. disclose wherein the circuitry is further to: continue a safety critical operation without interruption after the hardware fault is detected and corrected (correcting the error and or forwarding the reference matrix to continue operation).
Referring to claim 4, in para. 0051 et al. disclose wherein the circuitry is further to: determine a location of a detected hardware fault in a matrix of the one or more matrices of the matrix operation hardware circuitry based at least in part on the one or more hardware checksums (the error value and location of the error are output from the checksum computation).
Referring to claim 5, in para. 0051, Mody et al. disclose wherein the circuitry is further to: generate metadata that indicates the determined location of the detected hardware fault (the error value and location of the error are output from the checksum computation).
Referring to claim 6, in Figure 7 and in para. 0041, Mody et al. disclose wherein the circuitry is further to: correct the detected hardware fault based at least in part on the generated metadata and the one or more hardware checksums (the difference between the computed checksum and the reference checksum can be used to determine the correct value).
Referring to claim 7, in para. 0037-0038, Mody et al. disclose safety critical operation defined by the Automotive Safety Integrity Levels (ASIL). In para. 0041, Mody et al. disclose wherein the circuitry is further to: continue a safety critical operation without interruption after the hardware fault is detected and corrected (correcting the error and or forwarding the reference matrix to continue operation).
Referring to claim 8, in para. 0043, Mody et al. disclose wherein the circuitry is further to: perform a checksum computation and a matrix multiplication on a same operand at a same data location in a matrix of the one or more matrices of the matrix operation hardware circuitry.
Referring to claim 9:
In Fig. 7, Mody et al. disclose an apparatus, comprising circuitry.
In para. 0040, Mody et al. disclose store a machine representation of data for a matrix W with dimensions of m by k, where m and k are both positive integer values (memory stores input matrices).
In para. 0040, Mody et al. disclose store a machine representation of data for a matrix X with dimensions of k by n, where n is a positive integer value (memory stores input matrices).
In para. 0040 and 0042, Mody et al. disclose store a machine representation of data for a matrix Y with dimensions of m by n (memory stores output matrix); perform a matrix operation on the matrix W and the matrix X and to store a result of the matrix operation in the matrix Y (dot product of matrices and memory stores output matrix).
In para. 0040 and 0042, Mody et al. disclose perform one or more checksum operations on one or more of the matrix W, the matrix X, and the matrix Y.
In para. 0040 and 0042, Mody et al. disclose detect a hardware fault in the circuitry based at least in part on a result of the one or more checksum operations (reference checksums and computed checksums used to find errors).
Referring to claim 10, in Figure 12 and in para. 0041-0042, Mody et al. disclose wherein the circuitry is further to: perform a column-wise checksum of the data for the matrix W; perform a row-wise checksum of the data for the matrix X; perform a column-wise checksum of the data for the matrix Y; and perform a row-wise checksum of the data for the matrix Y.
Referring to claim 11, in Figure 12 A-C and in para. 0042, Mody et al. disclose wherein the circuitry is further to: detect the hardware fault based at least in part on whether a product of the column-wise checksum of the data for the matrix W and a data value in the matrix X is equal to a corresponding value of the column-wise checksum of the data for the matrix Y (matrix multiplication produces a matrix with column checksums compared with checksums of reference matrix).
Referring to claim 12, in Figure 12 A-C and in para. 0042, Mody et al. disclose wherein the circuitry is further to: detect the hardware fault based at least in part on whether a product of the row-wise checksum of the data for the matrix X and a data value in the matrix W is equal to a corresponding value of the row-wise checksum of the data for the matrix Y (matrix multiplication produces a matrix with row checksums compared with checksums of reference matrix).
Referring to claim 13, in Figure 6C and para. 0043, Mody et al. disclose wherein the circuitry is further to: detect the hardware fault based at least in part on whether a product of the column-wise checksum of the data for the matrix W and the row-wise checksum of the data for the matrix X is equal to a sum of the row-wise checksum and the column-wise checksum of the data for the matrix Y.
Referring to claim 15:
In Figure 7, Mody et al. disclose an apparatus, comprising: a processor; and a matrix compute engine communicatively coupled to the processor, the matrix compute engine comprising circuitry.
In para. 0036, Mody et al. disclose perform a matrix operation on a first input matrix and a second input matrix and store respective results of the matrix operation in an output matrix (matrix multiplication).
In para. 0041-0042, Mody et al. disclose perform respective checksum operations on one or more row and columns of the first input matrix (Checksum Matrix A), the second input matrix (Checksum Matrix B), and the output matrix (Checksum result—checksum matrix C).
On page 175, Section III, part A, Zhu et al. disclose and detect a hardware fault in the matrix compute engine based at least in part on respective results of the respective checksum operations (To check if there is an error during the multiplication, we just need to check w+ t pairs of data. The error block can be located at the intersection of mismatching block checksum).
Referring to claim 17, in Figure 7 and in para. 0041, Mody et al. disclose wherein the circuitry is further to: correct the detected hardware fault based at least in part on respective comparisons of the respective results of the respective checksum operations.
Referring to claim 18, in para. 0037-0038, Mody et al. disclose safety critical operation defined by the Automotive Safety Integrity Levels (ASIL). In para. 0041, Mody et al. disclose wherein the circuitry is further to: continue a safety critical operation without interruption after the hardware fault is detected and corrected (correcting the error and or forwarding the reference matrix to continue operation).
Referring to claim 19, in para. 0051 et al. disclose wherein the circuitry is further to: determine a location of a detected hardware fault in one or more of the first input matrix and the second input matrix based at least in part on respective comparisons of the respective results of the respective checksum operations (the error value and location of the error are output from the checksum computation)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 8, 9, 10, 14, 15, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al., “Block-checksum-based Fault Tolerance for Matrix Multiplication on Large-Scale Parallel systems”, and further in view of Mody et al., US 2020/0074287 A1.
Referring to claim 1:
On page 172, col. 2, Zhu et al. disclose matrix multiplication on large-scale parallel systems. However, Zhu et al. do not explicitly disclose an apparatus, comprising: matrix operation hardware circuitry and circuitry to detect a hardware fault based on hardware checksums. In Figures 4 and 6, Mody et al. disclose circuitry for performing matrix operations and checksums and memory for storing matrices. It would have been obvious to one of ordinary skill at the time of filing of the invention to include the circuitry and memory of Mody et al. into the system of Zhu et al. A person of ordinary skill in the art would have been motivated to make the modification because the hardware shown in Mody et al. is capable of performing the operations and functions disclosed by Zhu et al. The hardware would be needed to implement the functionality of Zhu et al. into the disclosed large-scale parallel system disclosed by Zhu et al.
On page 173, Section II, part B, Zhu et al. disclose circuitry coupled to the matrix operation hardware circuitry to detect a hardware fault in the matrix operation hardware circuitry based at least in part on one or more hardware checksums of data in one or more matrices of the matrix operation hardware circuitry (column checksum of matrix A and row checksum of matrix B is used to determine the error element in the matrix C after matrix multiplication).
Referring to claim 4, on page 173, Section II, part B, Zhu et al. disclose wherein the circuitry is further to: determine a location of a detected hardware fault in a matrix of the one or more matrices of the matrix operation hardware circuitry based at least in part on the one or more hardware checksums (The error element can be located at the intersection of the row and the column with mismatched checksums—page 173, col. 2).
Referring to claim 8, on page 173, Section II, part B, Zhu et al. disclose wherein the circuitry is further to: perform a checksum computation and a matrix multiplication on a same operand at a same data location in a matrix of the one or more matrices of the matrix operation hardware circuitry (column checksum on matrix A and row checksum on matrix B and multiply matrix A and B to obtain matrix C).
Referring to claim 9:
On page 172, col. 2, Zhu et al. disclose matrix multiplication on large-scale parallel systems. However, Zhu et al. do not explicitly disclose an apparatus, comprising: circuitry to store a machine representation of data for the matrices, perform matrix operations, perform checksum operations, and detect a hardware fault based on a checksum. In Figures 4 and 6, Mody et al. disclose circuitry for performing matrix operations and checksums and memory for storing matrices. It would have been obvious to one of ordinary skill at the time of filing of the invention to include the circuitry and memory of Mody et al. into the system of Zhu et al. A person of ordinary skill in the art would have been motivated to make the modification because the hardware shown in Mody et al. is capable of performing the operations and functions disclosed by Zhu et al. The hardware would be needed to implement the functionality of Zhu et al. into the disclosed large-scale parallel system.
On page 173, Section II, part B, Zhu et al. disclose store a machine representation of data for a matrix W with dimensions of m by k, where m and k are both positive integer values (column checksum matrix A with size (m+1) x n).
On page 173, Section II, part B, Zhu et al. disclose store a machine representation of data for a matrix X with dimensions of k by n, where n is a positive integer value (row checksum matrix B with size n x (q+1)).
On page 173, Section II, part B, Zhu et al. disclose store a machine representation of data for a matrix Y with dimensions of m by n; perform a matrix operation on the matrix W and the matrix X and to store a result of the matrix operation in the matrix Y (C checksum = A column checksum x B row check sum with size (m+1) x (q +1)).
On page 173, Section II, part B, Zhu et al. disclose perform one or more checksum operations on one or more of the matrix W, the matrix X, and the matrix Y.
On page 173, Section II, part B, Zhu et al. disclose detect a hardware fault in the circuitry based at least in part on a result of the one or more checksum operations (column checksum of matrix A and row checksum of matrix B is used to determine the error element in the matrix C after matrix multiplication).
Referring to claim 10, on page 173, Section II, part B, Zhu et al. disclose wherein the circuitry is further to: perform a column-wise checksum of the data for the matrix W (encode matrix A into a column checksum); perform a row-wise checksum of the data for the matrix X (encode matrix B into a row checksum); perform a column-wise checksum of the data for the matrix Y; and perform a row-wise checksum of the data for the matrix Y (matrix C is a column and row checksum matrix).
Referring to claim 14, on page 173, Section II, part B, Zhu et al. wherein the circuitry is further to: determine a location of a fault in the matrix Y based on respective positions of failed comparisons for the row-wise checksum and the column-wise checksum of the data for the matrix Y (The error element can be located at the intersection of the row and the column with mismatched checksums—page 173, col. 2).
Referring to claim 15:
On page 172, col. 2, Zhu et al. disclose matrix multiplication on large-scale parallel systems. However, Zhu et al. do not explicitly disclose an apparatus, comprising: a processor and a matrix compute engine. In Figures 4 and 6, Mody et al. disclose circuitry for performing matrix operations and checksums and memory for storing matrices. It would have been obvious to one of ordinary skill at the time of filing of the invention to include the circuitry and memory of Mody et al. into the system of Zhu et al. A person of ordinary skill in the art would have been motivated to make the modification because the hardware shown in Mody et al. is capable of performing the operations and functions disclosed by Zhu et al. The hardware would be needed to implement the functionality of Zhu et al. into the disclosed large-scale parallel system.
On page 174, Section III, part A, Zhu et al. disclose perform a matrix operation on a first input matrix and a second input matrix and store respective results of the matrix operation in an output matrix (matrix multiplication C=AxB).
On page 174, Section III, part A, Zhu et al. disclose perform respective checksum operations on one or more row and columns of the first input matrix (Checksum Matrix A), the second input matrix (Checksum Matrix B), and the output matrix (Checksum result—checksum matrix C).
On page 175, Section III, part A, Zhu et al. disclose and detect a hardware fault in the matrix compute engine based at least in part on respective results of the respective checksum operations (To check if there is an error during the multiplication, we just need to check w+ t pairs of data. The error block can be located at the intersection of mismatching block checksum).
Referring to claim 20, on page 174, Section III, part A, Zhu et al. disclose wherein the circuitry is further to: perform one or more of the respective checksum operations in parallel with the matrix operation (the multiplication is parallelly executed).
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mody et al., US 2020/0074287 A1 as applied to claim15 above, and further in view of Liu et al., US 2023/0108629 A1.
Referring to claim 16, in para. 0036, Mody et al. disclose neural network operations may be computed by performing matrix operations on the inputs to simulate the neurons in the various layers and produce the outputs. However, Mody et al. do not explicitly disclose circuitry to flatten and rearrange weights of a three-dimensional convolution to map the three-dimensional convolution to multiple matrix multiplication operations.
In para. 0031 and 0036, Liu et al. disclose a convolutional layer and a pooling layer may form a single layer of a CNN. The fully-connected layers follow the convolutional and pooling layers, and include a flatten layer and a classification layer, followed by a normalization layer that includes a normalization function, such as the SoftMax function. In para. 0039 and 0042, Liu et al. disclose training a CNN includes optimizing the connection weights between nodes by minimizing the prediction error of the output data until the CNN achieves a particular level of accuracy.
In para. 0069, Liu et al. disclose the convolutional layer calculations for CNNs may be converted into generic matrix multiplication (GEMM) operations for processing by one or more MMAs. Convolution layer calculations converted into a GEMM operation by converting filters into converted weight matrix, converting input feature maps into converted input data matrix, and then multiplying converted weight matrix and converted input data matrix to generate converted output data matrix. It would have been obvious to one of ordinary skill at the time of filing of the invention to include the flatten operation and conversion of convolution calculations to matrix multiplications of Liu et al. into the neural network system of Mody et al. A person of ordinary skill in the art would have been motivated to make the modification because Mody et al. disclose a neural network and performing matrix multiplication for the neural network. Further, the convolution operations are simplified because simple matrix multiplication is performed rather than a convolution operation. Each output element within converted output data matrix is the dot product of one row of converted weight matrix and one column of converted input data matrix (see Liu et al.: para. 0069).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shi et al. disclose performing checksums with matrix multiplication
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C MASKULINSKI whose telephone number is (571)272-3649. The examiner can normally be reached Monday-Friday 8:00 am-5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bryce Bonzo can be reached at (571) 272-3655. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL MASKULINSKI/Primary Examiner, Art Unit 2113
Prosecution Insights