Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on June 26, 2024 and on November 18, 2024 were filed and considered by the examiner. The submission is in compliance with the provisions of 37 CFR 1.97.
Claim Objections
Claim 16 is objected to under 37 CFR 1.75 as being a substantial duplicate of claim 14. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
The term "lower" from “is performed with a lower priority” in claim 13 is a relative term which renders the claim indefinite. The term "lower" is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process) without significantly more.
Claim 1:
Regarding claim 1, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites
“A method of processing a multilayer neural network with a neural network processor by tapering in weight matrix data from a memory coupled to said neural network processor, said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; each said subset of neural network layers referred to as a partition; dividing each neural network layer of each said partition into a set of work fragments, each work fragment comprising a subset of computations for said neural network layer; grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously; loading into said neural network processor a first work fragment subset for a first partition from said memory; loading in a first subset of weight matrix data from said external memory for said first work fragment subset of said first partition into said neural network processor; commencing processing of said first work fragment subset when said first subset of weight matrix data is available; loading in a second subset of weight matrix data from said external memory, if not already loaded, for a second work fragment subset for said first partition into said neural network processor while processing said first work fragment subset for said first partition; loading said second work fragment subset for said first partition into said neural network processor from said external memory; and processing said second work fragment subset for said first partition, when said second subset of weight matrix data for said second work fragment subset is available,” and a method is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; each said subset of neural network layers referred to as a partition; (This is considered a mental process, a person can mentally evaluate and divide a multilayer neural network into subsets, see MPEP 2106.04(a)(2)(III)),
dividing each neural network layer of each said partition into a set of work fragments, each work fragment comprising a subset of computations for said neural network layer; (This is considered a mental process, a person can mentally evaluate and divide each neural network layer into work fragments, see MPEP 2106.04(a)(2)(III)),
grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously; (This is considered a mental process, a person can mentally evaluate and group work fragments into subsets, see MPEP 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
A method of processing a multilayer neural network with a neural network processor by tapering in weight matrix data from a memory coupled to said neural network processor, (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
loading into said neural network processor a first work fragment subset for a first partition from said memory; (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
loading in a first subset of weight matrix data from said external memory for said first work fragment subset of said first partition into said neural network processor; (In step 2A, prong 2, loading recites mere data inputting and receiving data, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
commencing processing of said first work fragment subset when said first subset of weight matrix data is available; (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
loading in a second subset of weight matrix data from said external memory, if not already loaded, for a second work fragment subset for said first partition into said neural network processor while processing said first work fragment subset for said first partition; (In step 2A, prong 2, loading recites mere data inputting and receiving data, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
loading said second work fragment subset for said first partition into said neural network processor from said external memory; (In step 2A, prong 2, loading recites mere data inputting and receiving data, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
and processing said second work fragment subset for said first partition, when said second subset of weight matrix data for said second work fragment subset is available, (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements iv, v, vii, and x recite mere instructions to apply the judicial exception using generic computer components, which are not indicative of significantly more. The additional elements vi, viii, and ix recite mere data gathering or outputting, and are considered insignificant extra-solution activities. In step 2B, these insignificant extra-solution activities are well understood routine and conventional activity which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)), as well as see court case Mayo, 566 U.S. at 79, 101 USPQ2d at 1968; OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1092-93 (Fed. Cir. 2015) (presenting offers and gathering statistics amounted to mere data gathering, see MPEP 2106.05(g)(3))).
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 2:
Regarding claim 2, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 2 also recites an additional element:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein said work fragment subsets contain work fragments from different neural network layers in said first partition, (In step 2A, prong 2, This is considered mere instructions to apply an exception using generic computer– see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 3:
Regarding claim 3, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 3 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein work fragments may be processed out of order such that a later neural network layer may be processed before an earlier neural network layer, (In step 2A, prong 2, performing an out of order processing of a multilayer neural network is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 4:
Regarding claim 4, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 4 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 further comprising: decompressing said first subset of weight matrix data loaded from said external memory, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer– see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 5:
Regarding claim 5, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 5 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein said first subset of weight matrix data may comprise one of several different data precisions, (In step 2A, prong 2, This is considered mere instructions to apply an exception using generic computer– see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 6:
Regarding claim 6, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 6 recites the following additional elements:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1, (In step 2A, prong 2, This is considered mere instructions to apply an exception using generic computer– see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
.. further comprising: reloading in said first subset of weight matrix data from said external memory after a context switch of said neural network processor, (In step 2A, prong 2, this recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g),). In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) – see MPEP 2106.05(d) (II)(i),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 7:
Regarding claim 7, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites
“ A method of processing a multilayer neural network with a neural network processor and tapering out weight matrix from said neural network processor, said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; said subsets of neural network layers referred to as partition; dividing each network layer in each partition into a set of work fragments; grouping set of said work fragments of each cut into work fragment subsets that can be processed simultaneously; loading into said neural network processor a first work fragment subset for a first partition from said external memory; loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition; commencing processing of said work fragments for said neural network layers of said first partition; discarding said first weight matrix for a first neural network fragment of said first partition after processing a final work fragment for said first network layer to free memory resources; and loading a second weight matrix for a neural network layer in a subsequent partition into said neural network processor while completing processing of said work fragments of said first partition,” and a method is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; said subsets of neural network layers referred to as partition; (This is considered a mental process, a person can mentally evaluate and divide a multilayer neural network into subsets of neural network layers, see MPEP 2106.04(a)(2)(III)),
dividing each network layer in each partition into a set of work fragments; (This is considered a mental process, a person can mentally evaluate and divide each network layer into a set of work fragments, see MPEP 2106.04(a)(2)(III)),
grouping set of said work fragments of each cut into work fragment subsets that can be processed simultaneously; (This is considered a mental process, a person can mentally evaluate and group set of work fragments into subsets, see MPEP 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
A method of processing a multilayer neural network with a neural network processor and tapering out weight matrix from said neural network processor, (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
loading into said neural network processor a first work fragment subset for a first partition from said external memory; loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition; (In step 2A, prong 2, this recites mere data inputting and receiving data, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
commencing processing of said work fragments for said neural network layers of said first partition; (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
discarding said first weight matrix for a first neural network fragment of said first partition after processing a final work fragment for said first network layer to free memory resources; (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
and loading a second weight matrix for a neural network layer in a subsequent partition into said neural network processor while completing processing of said work fragments of said first partition, (In step 2A, prong 2, this recites mere data inputting and receiving data, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements iv, vi, and vii, recite mere instructions to apply the judicial exception using generic computer components, which are not indicative of significantly more. The additional elements v and viii recite mere data gathering or outputting, and are considered insignificant extra-solution activities. In step 2B, these insignificant extra-solution activities are well understood routine and conventional activities which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)), as well as see court case Mayo, 566 U.S. at 79, 101 USPQ2d at 1968; OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1092-93 (Fed. Cir. 2015) (presenting offers and gathering statistics amounted to mere data gathering, see MPEP 2106.05(g)(3))).
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 8:
Regarding claim 8, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 8 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and tapering weight matrix data from said neural network processor as set forth in claim 7, said method further comprising: decompressing said first weight matrix loaded from said external memory, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 9:
Regarding claim 9, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 9 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 wherein said first weight matrix may comprise one of several different data precisions, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 10:
Regarding claim 10, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 10 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 further comprising: reloading in said first weight matrix from said external memory after a context switch of said neural network processor, (In step 2A, prong 2, this recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g),). In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) – see MPEP 2106.05(d) (II)(i),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 11:
Regarding claim 11, it is dependent upon claim 7, and thereby incorporates the limitations of, and corresponding analysis applied to claim 7. Further, claim 11 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 wherein said partitions can belong to different neural networks, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 12:
Regarding claim 12, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory, said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; said subsets of neural network layers referred to as partition; dividing each network layer in each partition into a set of work fragments; grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously; loading into said neural network processor a first work fragment subset for a first partition from said external memory; loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition; commencing processing of said work fragments for said neural network layers of said first partition; prefetching a second weight matrix for a neural network layer in a subsequent partition from said external memory into said neural network processor while processing of said work fragments of said first partition when memory bandwidth is available to said external memory; and storing said second weight matrix in said neural network processor until said subsequent partition is triggered and said second weight matrix is needed for processing,” and a method is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; said subsets of neural network layers referred to as partition; (This is a mental process, a person can mentally evaluate and divide a multilayer neural network into subsets, see MPEP 2106.04(a)(2)(III)),
dividing each network layer in each partition into a set of work fragments; (This is a mental process, a person can mentally evaluate and divide each network layer, see MPEP 2106.04(a)(2)(III)),
grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously; (This is a mental process, a person can mentally evaluate and group work fragments, see MPEP 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
A method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory, (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
loading into said neural network processor a first work fragment subset for a first partition from said external memory; (In step 2A, prong 2, loading recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition; (In step 2A, prong 2, loading recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
commencing processing of said work fragments for said neural network layers of said first partition; (This is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
prefetching a second weight matrix for a neural network layer in a subsequent partition from said external memory into said neural network processor while processing of said work fragments of said first partition when memory bandwidth is available to said external memory; (In step 2A, prong 2, prefetching, which is similar to loading, recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
and storing said second weight matrix in said neural network processor until said subsequent partition is triggered and said second weight matrix is needed for processing, (In step 2A, prong 2, this recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional element iv and vii recite mere instructions to apply the judicial exception using generic computer components, which are not indicative of significantly more. The additional elements v, vi, viii, and ix recite mere data gathering, and are considered insignificant extra-solution activities. In step 2B, these insignificant extra-solution activities are well understood routine and conventional activities which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)),
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 13:
Regarding claim 13, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis applied to claim 12. Further, claim 13 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said prefetching is performed with a lower priority than other accesses to said external memory, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 14:
Regarding claim 14, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis applied to claim 12. Further, claim 14 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said partitions can belong to different neural networks, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 15:
Regarding claim 15, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis applied to claim 12. Further, claim 15 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said work fragments can be executed out of order, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 16:
Regarding claim 16, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis applied to claim 12. Further, claim 16 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said partitions can belong to different neural networks, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 17:
Regarding claim 17, it is dependent upon claim 12, and thereby incorporates the limitations of, and corresponding analysis applied to claim 12. Further, claim 17 recites the following additional element:
The method of processing a multilayer neural network with a neural network processor and tapering weight matrix data out from said neural network processor as set forth in claim 12, said method further comprising: decompressing said second weight matrix prefetched from said external memory, (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)), (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 12 are rejected under 35 U.S.C. 103 over TIMOFEJEVS, A. et al. (Pub. No. WO 2021259482 A1), published on December 30, 2021, (hereafter, TIMOFEJEVS),
in view of Le Grand, S. et al. (Pub. No. WO2017117186A1), published on July 6, 2017, (hereafter, LEGRAND), and further in view of Fang, W. et al., (Pub. No. CN112667528A), published on April 16, 2021, (hereafter, FANG).
Claim 1:
Regarding claim 1, TIMOFEJEVS teaches “A method of processing a multilayer neural network with a neural network processor by tapering in weight matrix data from a memory coupled to said neural network processor, said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; each said subset of neural network layers referred to as a partition;”
See TIMOFEJEVS in paragraph [0016] describe "In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network." Note, the examiner construes the word partition to mean any subdivision of a neural network model. Here, TIMOFEJEVS describes an implementation that sub-divides a neural network first from layers to sub-layers, then from sub-layers into sub-function blocks. TIMOFEJEVS in paragraph [0016] mentioned “Each sub-layer implements an intermediate mathematical function”, relates to (i.e. each subset will be processed as a group ). Regarding the subset of neural network layers referred to as a partition limitation, the instant application’s specification paragraph [0085] states “each partition of neural network layers may be processed together as a group.”
Further, see TIMOFEJEVS in paragraph [0037] mention “in some implementations, the neural network topology includes K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons.” Note, tapering according to specification paragraphs [0092-0093] is interpreted to mean the same as loading or inputting in data. Here, in [0037], TIMOFEJEVS mentions the neural network includes inputs, which shows loading in data that includes weight matrix values.
Further, see TIMOFEJEVS in paragraph [0145] “2. If K>N then: a. Divide K input neurons into m1 =
PNG
media_image1.png
103
75
media_image1.png
Greyscale
groups such that every group consists of no more than N inputs. b. Construct the first hidden layer LTHi of the T-NN from rr^neurons, each neuron performing an identity activation function. c. Connect input neurons from every group to corresponding neuron from the next layer.” Here, TIMOFEJEVS describes a part of the neural network that handles data is treated as a group or subdivision (i.e. each said subset of neural network layers referred to as a partition) See TIMOFEJEVS for more information in paragraphs [0142-0143] and [00282].
Further, TIMOFEJEVS teaches “dividing each neural network layer of each said partition into a set of work fragments, each work fragment comprising a subset of computations for said neural network layer;”
See TIMOFEJEVS in paragraph [0016] describe "in some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network." Here, TIMOFEJEVS shows the sub-function blocks as the work fragments from “selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer” (i.e. dividing each neural network layer of each said partition into a set of work fragments). This shows that each sub-function blocks have its own math function and calculations associated with its respective task, and further indicates the segmented nature of a neural network.
However, TIMOFEJEVS did not teach
“grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously;”
“ loading into said neural network processor a first work fragment subset for a first partition from said memory;”
“loading in a first subset of weight matrix data from said external memory for said first work fragment subset of said first partition into said neural network processor;”
“commencing processing of said first work fragment subset when said first subset of weight matrix data is available,”
“loading in a second subset of weight matrix data from said external memory, if not already loaded, for a second work fragment subset for said first partition into said neural network processor while processing said first work fragment subset for said first partition;”
“loading said second work fragment subset for said first partition into said neural network processor from said external memory;”
“and processing said second work fragment subset for said first partition, when said second subset of weight matrix data for said second work fragment subset is available,”
In an analogous art, LEGRAND teaches “grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously;”
See LEGRAND in paragraph [0020] describe “as shown in FIG. 1, however, the weight matrix 120 may be split among a plurality of different computer processors, and the processors may generate different portions of the matrix 130 in parallel. For example, the weight matrix 120 may be striped row-wise (separated into subsets of rows), and each processor may be provided with a different subset of the rows. The input matrix 110 may be striped column-wise (separated into subsets of columns), and each processor may be provided with a different subset of the columns. … In some embodiments, the matrix 130 may be generated by performing a series of "reduction" operations or some equivalent operation in which multiple sets of numbers— the intermediate matrices in this example— are reduced into a single set of numbers— the subset of columns of matrix 130 to be stored on an individual processor. A reduction operation can be performed to aggregate, from the intermediate matrices, each separate subset of columns to be stored on each individual processor. In some cases, the reduction operations may be performed substantially in parallel or otherwise at least partially overlapping in time.” Here, LEGRAND shows that each processor has different portions of data named matrix 130. Then, LEGRAND describes that grouping different portions of the matrix data (i.e. relate to partition) with reduction operations being a work fragment, that may be performed substantially in parallel or otherwise at least partially overlapping in time (i.e. work fragment of each partition processed simultaneously). Note the examiner construes work fragment to mean any task, job, part of a data, that is performed on data.
Further, see LEGRAND in paragraph [0004] for more details “FIG. 1 is a diagram of an illustrative artificial neural network with multiple layers, indicating how the layers are to be distributed among multiple computer processors for parallel processing.” LEGRAND further elaborates that the process described in [0020] is part of neural networks with multiple layers, and each layer needs a number of processors to process data in parallel or simultaneously.
Further, LEGRAND teaches “loading into said neural network processor a first work fragment subset for a first partition from said memory;”
See LEGRAND in paragraph [0028] describe “..the individual computer processors multiply their own subsets of columns of the current matrix by their own subsets of rows of the current weight matrix. In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors.” See LEGRAND in paragraph [0023] for more details on loading from memory.
Further, see LEGRAND describe in [0002] “Sets of individual input vectors ("mini-batches") may be processed at the same time by using an input matrix instead of a single input vector. The NN can repeatedly process the input data, and the parameters (e.g., the weight matrices) of the NN can be modified in what amounts to a trial-and-error process until the model produces (or "converges" on) the correct or preferred output.” LEGRAND describes this can be done repeatedly for input data of the neural network, … and includes loading data for any work fragment for any layer or partition of the neural network.
Later in paragraph [0054] Clause 1. LEGRAND describes “a system comprising a plurality of processors, the system programmed by executable instructions to at least: obtain data defining an artificial neural network, the artificial neural network comprising a first layer of nodes, a second layer of nodes, and a third layer of nodes, wherein the first layer comprises more nodes than the second layer, and wherein the third layer comprises more nodes than the second layer; provide to a first processor of the plurality of processors: a first column of input data from a first data matrix, the first data matrix comprising input data for the artificial neural network; a first row of weights from a first weight matrix, the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer; and a first column of weights from a second weight matrix, the second weight matrix comprising weights for connections between nodes of the second layer and nodes of the third layer; provide to a second processor of the plurality of processors: a second column of input data from the first data matrix; a second row of weights from the first weight matrix; and a second column of weights from the second weight matrix; compute, using the first processor, a first subset of columns of a second data matrix of values for the second layer, wherein the first subset is computed from the first column of input data, the first row of weights, and aggregated values received from the second processor of the plurality of processors”. Note the examiner construes loading data to mean similar to obtaining data. Here, LEGRAND mentions loading into a processor of the neural network and specifies the obtaining or loading of the column of weights (i.e. first work fragment) for a first layer of the first subset (i.e. first partition) of a neural network.
Further, see LEGRAND in [0054] Clause 5, for more detail, describe “computing, by the plurality of computer processors, the second layer matrix, wherein individual computer processors of the plurality of computer processors each compute a different contribution to the second layer matrix using a corresponding subset of columns of the first layer matrix and a corresponding subset of rows of a first weight matrix, and wherein the first weight matrix comprises weights for connections between nodes of the first layer and nodes of the second layer;” See LEGRAND for more information in paragraphs [0011] and [0018].
Further, LEGRAND teaches “loading in a first subset of weight matrix data from said external memory for said first work fragment subset of said first partition into said neural network processor,”
See LEGRAND in paragraph [0043] describe “FIG. 5 and otherwise described herein, a computing system 500 may include various other components, such as one or more network interfaces (e.g., network interface cards), one or more computer readable medium drives (e.g., high density disks, solid state drives, flash drives, and/or other persistent non-transitory computer-readable media), an input/output device interface (e.g. an 10 interface in communication with one or more microphones or display screens), and one or more computer readable memories (e.g., random access memory and/or other volatile non-transitory computer-readable media).” Note the examiner construes external memory to mean any storage device used for long-term data storage, such as flash drives. Here, LEGRAND describes devices used for long-term storage for external memory.
Later, see LEGRAND in paragraph [0054] Clause 1. describe “a system comprising a plurality of processors, the system programmed by executable instructions to at least: obtain data defining an artificial neural network, the artificial neural network comprising a first layer of nodes, a second layer of nodes, and a third layer of nodes, wherein the first layer comprises more nodes than the second layer, … provide to a first processor of the plurality of processors: a first column of input data from a first data matrix, the first data matrix comprising input data for the artificial neural network; a first row of weights from a first weight matrix, the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer; and a first column of weights from a second weight matrix, the second weight matrix comprising weights for connections between nodes of the second layer and nodes of the third layer; provide to a second processor of the plurality of processors: a second column of input data from the first data matrix; a second row of weights from the first weight matrix; and a second column of weights from the second weight matrix; compute, using the first processor, a first subset of columns of a second data matrix of values for the second layer, wherein the first subset is computed from the first column of input data, the first row of weights, and aggregated values received from the second processor of the plurality of processors”. Here, when LEGRAND mentions obtaining “the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer” into a processor within a neural network, this shows loading in a first subset of weight matrix data from said external memory for said first work fragment subset of said first partition into said neural network processor, where the weights for connections between nodes of first and second layers are the first subset.
Further, LEGRAND teaches “commencing processing of said first work fragment subset when said first subset of weight matrix data is available;”
See LEGRAND in paragraph [0028] describe “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors. This may happen before processing input matrices because the structure of the NN 100 (the sizes of the respective layers, and therefore the sizes of the input and output matrices for each NN operation) is predetermined”. Here, LEGRAND mentions the processing of any part of the input matrices (i.e. work fragment subset) starts when the weight matrices are made accessible (i.e. available) to the corresponding computer processors. Note the examiner construes available to mean accessible or in a ready to process format. See LEGRAND in paragraph [0036] for more details.
Further, LEGRAND teaches “loading said second work fragment subset for said first partition into said neural network processor from said external memory;”
See LEGRAND in paragraph [0028] mention “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors.” Here, in [0028], LEGRAND shows the relevant subsets and work fragments of the neural network have been loaded onto the processor from memory.
Further, see LEGRAND in paragraph [0054] Clause 5, describe “computing, by the plurality of computer processors, the second layer matrix, wherein individual computer processors of the plurality of computer processors each compute a different contribution to the second layer matrix using a corresponding subset of columns of the first layer matrix and a corresponding subset of rows of a first weight matrix, and wherein the first weight matrix comprises weights for connections between nodes of the first layer and nodes of the second layer;” Here, LEGRAND shows the second layer matrix is part of the second work fragment subset and is used to compute information that correspond to a subset of columns from the first layer matrix, (i.e. loading second work fragment subset for said first partition). See LEGRAND for more information in paragraphs [0011] and [0018].
Further, LEGRAND teaches “and processing said second work fragment subset for said first partition, when said second subset of weight matrix data for said second work fragment subset is available,”
See LEGRAND in paragraph [0028] mention “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors.” Here, in [0028], LEGRAND shows the relevant subsets and work fragments of the neural network have been loaded onto the processor from memory, is accessible (i.e. available), and is being processed by processors.
Further, see LEGRAND in paragraph [0054] Clause 5, describe “computing, by the plurality of computer processors, the second layer matrix, wherein individual computer processors of the plurality of computer processors each compute a different contribution to the second layer matrix using a corresponding subset of columns of the first layer matrix and a corresponding subset of rows of a first weight matrix, and wherein the first weight matrix comprises weights for connections between nodes of the first layer and nodes of the second layer;” Here, LEGRAND shows the second layer matrix is part of the second work fragment subset and is used to compute information that correspond to a subset of columns from the first layer matrix, (i.e. loading second work fragment subset for said first partition). See LEGRAND for more information in paragraphs [0011] and [0018].
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of TIMOFEJEVS and incorporate into the teachings of LEGRAND because both references teach processing a neural network into partitions and subsets.
One of ordinary skill in the art would be motivated to do so because these methods “provide efficient parallelization for artificial neural network processing by minimizing or otherwise reducing the communication required between individual computer processors for performing various artificial neural network operations in parallel. Such efficiently parallelized artificial neural networks may be used in a variety of machine learning applications and other systems, including but not limited to: product recommendation generation, automatic speech recognition, facial recognition, handwriting recognition, and image recognition” (LEGRAND, [0010]).
However, TIMOFEJEVS in view of LEGRAND did not teach “loading in a second subset of weight matrix data from said external memory, if not already loaded, for a second work fragment subset for said first partition into said neural network processor while processing said first work fragment subset for said first partition;”
In an analogous art, FANG teaches “loading in a second subset of weight matrix data from said external memory, if not already loaded, for a second work fragment subset for said first partition into said neural network processor while processing said first work fragment subset for said first partition;”
See FANG describe in paragraph [n0047] that “Prefetching refers to loading data from main memory (such as hard disk) into cache memory (cache) in advance before the CPU accesses the data, in order to reduce the pause time when the CPU accesses data and improve the CPU performance.” Here, FANG mentions that prefetching means the same as loading in data.
Further, see FANG in paragraph [n0075] “For example, if the calculated value of 'a' is 0.3 and the value of 'b' is 0.7, and the computing device needs to prefetch the data corresponding to 10 associated addresses into the cache each time, the computing device will select 3 first associated addresses from all the first associated addresses predicted and output by the first neural network model and prefetch the corresponding data into the cache; at the same time, the computing device will select 7 second associated addresses from all the second associated addresses predicted and output by the second neural network model and prefetch the corresponding data into the cache.”
Further, see FANG in [n0102] describe “Then, the increment ΔW2 of the weight matrix W2 is calculated using h and v, i.e., ΔW2 = h * v. The calculated ΔW2 is used to correct W2, that is, the corrected weight matrix of the output layer = W2 + α*ΔW2, where α represents the learning rate, which ranges from 0 to 1 and can be set as needed.
Calculate the increment ΔW1 of the weight matrix W1 using x, W2, and v, i.e., ΔW1 = x * (W2 * e).”
Later, see FANG in paragraph [n0108] mention “After training is completed on one training sample in the training set, that is, after training is completed on one set of addresses, training is performed on another training sample to continue to correct (update) W1 and W2. The specific training process is the same as described above. It should be noted that the samples in the training set may be used for training multiple times, not just once.” FANG mentions that W1 and W2 are the weight matrices for each neural network model. FANG shows loading or prefetching using a second weight matrix W2 from cache (i.e. second subset of weight matrix data from memory) for a training on the training set which may be used for training multiple times (i.e. second work fragment subset) for a first group of a neural network while continue to update W1 (i.e. while processing a first work fragment subset).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of TIMOFEJEVS, and LEGRAND and incorporate into the teachings of FANG by using the teachings of TIMOFEJEVS and LEGRAND of dividing a neural network into subsets with FANG’s teaching of loading in a second subset of weight matrix data from said external memory, if not already loaded, for a second work fragment subset for said first partition into said neural network processor while processing said first work fragment subset for said first partition.
One of ordinary skill in the art would be motivated to do so because by integrating FANG’s framework into the methods of TIMOFEJEVS and LEGRAND, one with ordinary skill in the art would achieve the goal of providing a method in “reducing the dimensionality can significantly reduce the amount of computation and improve training efficiency,” (FANG, [n0096]), and "if the data accessed by the user can be found in the cache, it is called a
cache hit; if it cannot be found, it is called a cache miss. To improve cache hit rate, it is necessary not only to rely on the local behavior of program execution and data access, but also to use data prefetching technology, which involves fetching the data that the CPU needs from the hard drive into the cache in advance," (FANG, paragraph [0004]).
Claim 12:
Regarding claim 12, TIMOFEJEVS teaches “A method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory, said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; said subsets of neural network layers referred to as partition;”
See TIMOFEJEVS in paragraph [0016] describe "In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network." Note, the examiner construes the word partition to mean any subdivision of a neural network model. Here, TIMOFEJEVS describes an implementation that sub-divides a neural network first from layers to sub-layers, then from sub-layers into sub-function blocks. TIMOFEJEVS in paragraph [0016] mentioned “Each sub-layer implements an intermediate mathematical function”, relates to (i.e. each subset will be processed as a group ). Regarding the subset of neural network layers referred to as a partition limitation, the instant application’s specification paragraph [0085] states “each partition of neural network layers may be processed together as a group.”
Further, see TIMOFEJEVS in paragraph [0145] “2. If K>N then: a. Divide K input neurons into m1 =
PNG
media_image1.png
103
75
media_image1.png
Greyscale
groups such that every group consists of no more than N inputs. b. Construct the first hidden layer LTHi of the T-NN from rr^neurons, each neuron performing an identity activation function. c. Connect input neurons from every group to corresponding neuron from the next layer.” Here, TIMOFEJEVS describes a part of the neural network that handles data is treated as a group or subdivision (i.e. each said subset of neural network layers referred to as a partition) See TIMOFEJEVS for more information in paragraphs [0142-0143] and [00282].
Further, TIMOFEJEVS teaches “dividing each network layer in each partition into a set of work fragments;”
See TIMOFEJEVS in paragraph [0016] describe "in some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network." Here, TIMOFEJEVS shows the sub-function blocks as the work fragments from “selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer” (i.e. dividing each neural network layer of each said partition into a set of work fragments). This shows that each sub-function blocks have its own math function and calculations associated with its respective task, and further indicates the segmented nature of a neural network.
However, TIMOFEJEVS did not teach:
“grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously;”
“loading into said neural network processor a first work fragment subset for a first partition from said external memory;”
“loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition;”
“commencing processing of said work fragments for said neural network layers of said first partition;”
“prefetching a second weight matrix for a neural network layer in a subsequent partition from said external memory into said neural network processor while processing of said work fragments of said first partition when memory bandwidth is available to said external memory;”
“and storing said second weight matrix in said neural network processor until said subsequent partition is triggered and said second weight matrix is needed for processing.”
In an analogous field, LEGRAND teaches “grouping set of said work fragments of each partition into work fragment subsets that can be processed simultaneously;”
See LEGRAND in paragraph [0020] describe “as shown in FIG. 1, however, the weight matrix 120 may be split among a plurality of different computer processors, and the processors may generate different portions of the matrix 130 in parallel. For example, the weight matrix 120 may be striped row-wise (separated into subsets of rows), and each processor may be provided with a different subset of the rows. The input matrix 110 may be striped column-wise (separated into subsets of columns), and each processor may be provided with a different subset of the columns. … In some embodiments, the matrix 130 may be generated by performing a series of "reduction" operations or some equivalent operation in which multiple sets of numbers— the intermediate matrices in this example— are reduced into a single set of numbers— the subset of columns of matrix 130 to be stored on an individual processor. A reduction operation can be performed to aggregate, from the intermediate matrices, each separate subset of columns to be stored on each individual processor. In some cases, the reduction operations may be performed substantially in parallel or otherwise at least partially overlapping in time.” Here, LEGRAND shows that each processor has different portions of data named matrix 130. Then, LEGRAND describes that grouping different portions of the matrix data (i.e. relate to partition) with reduction operations being a work fragment, that may be performed substantially in parallel or otherwise at least partially overlapping in time (i.e. work fragment of each partition processed simultaneously). Note the examiner construes work fragment to mean any task, job, part of a data, that is performed on data.
Further, see LEGRAND in paragraph [0004] for more details “FIG. 1 is a diagram of an illustrative artificial neural network with multiple layers, indicating how the layers are to be distributed among multiple computer processors for parallel processing.” LEGRAND further elaborates that the process described in [0020] is part of neural networks with multiple layers, and each layer needs a number of processors to process data in parallel or simultaneously.
Further, LEGRAND teaches “loading into said neural network processor a first work fragment subset for a first partition from said external memory;”
See LEGRAND in paragraph [0028] describe “..the individual computer processors multiply their own subsets of columns of the current matrix by their own subsets of rows of the current weight matrix. In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors.” LEGRAND mentions subsets of data are loaded onto corresponding processors. See LEGRAND in paragraph [0023] for more details on loading from memory.
Further, see LEGRAND describe in [0002] “Sets of individual input vectors ("mini-batches") may be processed at the same time by using an input matrix instead of a single input vector. The NN can repeatedly process the input data, and the parameters (e.g., the weight matrices) of the NN can be modified in what amounts to a trial-and-error process until the model produces (or "converges" on) the correct or preferred output.” LEGRAND describes this can be done repeatedly for input data of the neural network, … and includes loading data for any work fragment for any layer or partition of the neural network.
Later in paragraph [0054] Clause 1. LEGRAND describes “a system comprising a plurality of processors, the system programmed by executable instructions to at least: obtain data defining an artificial neural network, the artificial neural network comprising a first layer of nodes, a second layer of nodes, and a third layer of nodes, wherein the first layer comprises more nodes than the second layer, and wherein the third layer comprises more nodes than the second layer; provide to a first processor of the plurality of processors: a first column of input data from a first data matrix, the first data matrix comprising input data for the artificial neural network; a first row of weights from a first weight matrix, the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer; and a first column of weights from a second weight matrix, the second weight matrix comprising weights for connections between nodes of the second layer and nodes of the third layer; provide to a second processor of the plurality of processors: a second column of input data from the first data matrix; a second row of weights from the first weight matrix; and a second column of weights from the second weight matrix; compute, using the first processor, a first subset of columns of a second data matrix of values for the second layer, wherein the first subset is computed from the first column of input data, the first row of weights, and aggregated values received from the second processor of the plurality of processors”. Note the examiner construes loading data to mean similar to obtaining data. Here, LEGRAND mentions loading into a processor of the neural network and specifies the obtaining or loading of the column of weights (i.e. first work fragment) for a first layer of the first subset (i.e. first partition) of a neural network.
Further, LEGRAND teaches “loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition;”
See LEGRAND in paragraph [0043] describe “FIG. 5 and otherwise described herein, a computing system 500 may include various other components, such as one or more network interfaces (e.g., network interface cards), one or more computer readable medium drives (e.g., high density disks, solid state drives, flash drives, and/or other persistent non-transitory computer-readable media), an input/output device interface (e.g. an 10 interface in communication with one or more microphones or display screens), and one or more computer readable memories (e.g., random access memory and/or other volatile non-transitory computer-readable media).” Note the examiner construes external memory to mean any storage device used for long-term data storage, such as flash drives. Here, LEGRAND describes devices used for long-term storage for external memory.
Later, see LEGRAND in paragraph [0054] Clause 1. describe “a system comprising a plurality of processors, the system programmed by executable instructions to at least: obtain data defining an artificial neural network, the artificial neural network comprising a first layer of nodes, a second layer of nodes, and a third layer of nodes, wherein the first layer comprises more nodes than the second layer, … provide to a first processor of the plurality of processors: a first column of input data from a first data matrix, the first data matrix comprising input data for the artificial neural network; a first row of weights from a first weight matrix, the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer; and a first column of weights from a second weight matrix, the second weight matrix comprising weights for connections between nodes of the second layer and nodes of the third layer; provide to a second processor of the plurality of processors: a second column of input data from the first data matrix; a second row of weights from the first weight matrix; and a second column of weights from the second weight matrix; compute, using the first processor, a first subset of columns of a second data matrix of values for the second layer, wherein the first subset is computed from the first column of input data, the first row of weights, and aggregated values received from the second processor of the plurality of processors”. Here, when LEGRAND mentions obtaining “the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer” into a processor within a neural network, this shows loading in a first subset of weight matrix data from said external memory for work fragments of a first partition into said neural network processor, where the weights for connections between nodes of first and second layers are the first subset. The first partition in this case is any part of the first layer of the neural network that LEGRAND shows.
Further, LEGRAND teaches “commencing processing of said work fragments for said neural network layers of said first partition;”
See LEGRAND in paragraph [0028] describe “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors. This may happen before processing input matrices because the structure of the NN 100 (the sizes of the respective layers, and therefore the sizes of the input and output matrices for each NN operation) is predetermined”. Here, LEGRAND mentions the processing of any part of the input matrices (i.e. work fragment subset) starts when the weight matrices are made accessible (i.e. available) to the corresponding computer processors. Note the examiner construes available to mean accessible or in a ready to process format. See LEGRAND in paragraph [0036] for more details.
Further, LEGRAND teaches “prefetching a second weight matrix for a neural network layer in a subsequent partition from said external memory into said neural network processor while processing of said work fragments of said first partition when memory bandwidth is available to said external memory;”
See LEGRAND in paragraph [0026] note that “at decision block 206, the computing system 500 can determine whether the current matrix (the input matrix 110 in this example) is larger than the next matrix to be computed (matrix 130 for the second layer 104 in this example). The determination may be based on one or more size-related characteristics of the respective matrices. For example, the determination may be based on how many elements are in each of the respective matrices, how many nonzero data values are in each of the respective matrices, how many rows are in each of the respective matrices, how many columns are in each of the respective matrices, how much bandwidth is required to transmit each of the respective matrices, how much time transmission of each of the respective matrices to each of the processors is expected to take, how much memory each of the respective matrices takes, some combination thereof, etc.” LEGRAND here mentions the decision block 206 evaluates the bandwidth that is needed to transmit matrix data and determine if memory is available to do this step (i.e. processing of said work fragments of said first partition when memory bandwidth is available to said external memory).
See LEGRAND in paragraph [0028] mention “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors.” Here, in [0028], LEGRAND shows the relevant subsets and work fragments of the neural network have been loaded onto the processor from memory, is accessible (i.e. available), and is being processed by processors.
Further, see LEGRAND in paragraph [0054] Clause 5, describe “computing, by the plurality of computer processors, the second layer matrix, wherein individual computer processors of the plurality of computer processors each compute a different contribution to the second layer matrix using a corresponding subset of columns of the first layer matrix and a corresponding subset of rows of a first weight matrix, and wherein the first weight matrix comprises weights for connections between nodes of the first layer and nodes of the second layer;” Here, LEGRAND shows the second layer matrix is part of the second work fragment subset and is used for calculations that correspond to a subset of columns from the first layer matrix.
Further, LEGRAND teaches “and storing said second weight matrix in said neural network processor until said subsequent partition is triggered and said second weight matrix is needed for processing.”
See LEGRAND in paragraph [0028] describe “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors. This may happen before processing input matrices …”. Here, LEGRAND mentions the processing of any part of the input matrices (i.e. work fragment subset) starts when the weight matrices are made accessible (i.e. available) to the corresponding computer processors. Note the examiner construes the term triggered to mean anything that starts a processor to start processing data. LEGRAND mentions the triggering here is related to when the process 200 is initiated.
Further, LEGRAND describes in paragraph [0019] “in a conventional system, an input matrix 110 would be provided to a computer processor that stores or otherwise has access to the entire weight matrix 120. The processor would then multiply the input matrix 110 by the weight matrix 120 to produce a matrix 130. The processor may adjust individual values in the matrix 130 using an offset or bias that is associated with the second layer 104 (e.g., by adding or subtracting a value separate from the weight that is applied). In addition, the processor may apply an activation function to the individual values in the matrix 130 (e.g., by using the individual values as input to sigmoid function). The matrix 130 may then serve as a second layer matrix, and may be the input matrix in a process to calculate values for the third layer 106.” Here, LEGRAND specifically teaches storing a weight matrix in the processor and then calculate values when needed to adjust individual values. LEGRAND mentions this weight matrix values are associated with the second layer of the neural network (i.e. second weight matrix).
Later, LEGRAND mentions in paragraph [0020] As shown in FIG. 1, however, the weight matrix 120 may be split among a plurality of different computer processors, and the processors may generate different portions of the matrix 130 in parallel... As described in greater detail below, the individual processors may then each generate an intermediate matrix by multiplying their respective subsets of input matrix 110 columns and weight matrix 120 rows. The processors may then generate the matrix 130 by communicating their respective intermediate matrices to the other processors,” LEGRAND describes here that the stored weight matrix data can then be used to generate respective intermediate matrices (where respective can be interpreted as first, second, or whichever partition is triggered) by communicating to the other processors, and this relates to a respective weight matrix is needed for processing.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of TIMOFEJEVS and incorporate into the teachings of LEGRAND because both references teach processing a neural network into partitions and subsets.
One of ordinary skill in the art would be motivated to do so because these methods “provide efficient parallelization for artificial neural network processing by minimizing or otherwise reducing the communication required between individual computer processors for performing various artificial neural network operations in parallel. Such efficiently parallelized artificial neural networks may be used in a variety of machine learning applications and other systems, including but not limited to: product recommendation generation, automatic speech recognition, facial recognition, handwriting recognition, and image recognition” (LEGRAND, [0010]).
However, LEGRAND did not explicitly teach “prefetching a second weight matrix for a neural network layer in a subsequent partition …;”
In an analogous method, FANG teaches “prefetching a second weight matrix for a neural network layer in a subsequent partition from said external memory into said neural network processor while processing of said work fragments of said first partition when memory bandwidth is available to said external memory;”
See FANG in paragraph [n0011] mention “In this embodiment of the application, when the processor stores data in the cache, it needs to first determine whether the data has already been stored in the cache. Only if the data is not stored in the cache will it be stored in the cache. This can avoid data duplication, reduce the waste of cache space, and improve the utilization rate of the cache.”
Later, see FANG in paragraph [n0004] describe “To improve cache hit rate, it is necessary not only to rely on the local behavior of program execution and data access, but also to use data prefetching technology, which involves fetching the data that the CPU needs from the hard drive into the cache in advance.” Here, FANG explicitly mentions using the prefetching method on data.
Further, see FANG in paragraphs [n0102-n0104] for more information. Also see [n0124] for more details in FANG.
Claim 2 is rejected under 35 U.S.C. over TIMOFEJEVS in view of LEGRAND, further in view of FANG, and further in view of Deisher, M. et al., (US PG Pub. No. US20180121796A1), published on May 3, 2018, (hereafter, DEISHER).
Claim 2:
Regarding claim 2, TIMOFEJEVS in view of LEGRAND, further in view of FANG teaches the limitations in claim 1.
However, TIMOFEJEVS in view of LEGRAND, further in view of FANG did not teach “the method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein said work fragment subsets contain work fragments from different neural network layers in said first partition.”
In an analogous art, DEISHER teaches “the method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein said work fragment subsets contain work fragments from different neural network layers in said first partition.”
See DEISHER in paragraph [0034] describes "regarding the activation function, the speed and efficiency of the NN processing may be further increased by the use of a highly efficient piecewise linear activation function that can efficiently support many different activation function types …, placing an activation function unit in a pipeline after the parallel logic structure produces a relatively high processing rate since the parallel logic may be performing computations by multiplying an input vector by values in a weight matrix for one output of a layer while the activation function is simultaneously computing a final output using a weighted input sum output from a different output (or node) of the same layer (or different layer)." Here, DEISHER mentions processing different layers within a neural network.
Further, see DEISHER in in paragraph [0116] describe" Also, the layer descriptors for certain format or arrangement layers may be inserted into the layer descriptor chain. This may include transpose layers that orient grouped (multiple output) data so that the data of individual outputs of a layer extends along a row of an input array for example when row by row processing is desired, or vice-versa when processing across multiple groups is desired ... A copy layer also may be inserted to copy layer data from one memory address location to another while concatenating multiple input vectors from different layers". Here, DEISHER shows input data including those from work fragment subsets, can originate from different layers of the neural network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of TIMOFEJEVS, LEGRAND, FANG and incorporate into the teachings of DEISHER by using the teachings of TIMOFEJEVS, LEGRAND, and FANG of dividing a neural network into subsets with DEISHER’s teaching of the subsets originating from different layers of a neural network.
One of ordinary skill in the art would be motivated to do so because by integrating DEISHER’s framework into the methods of TIMOFEJEVS, LEGRAND, and FANG, one with ordinary skill in the art would achieve the goal of providing a method that “using the highly parallel logic structure of the NNA … may include the use of flexible layer descriptor definitions that enable many complex neural networks with variations such as topologies, operations, and sizes to be defined as a set of supported … layer types that further increases the efficiency of the processing of the layers. The flexible layer descriptor definitions enable a very adaptable neural network accelerator that can be used for many different neural network types and layer sequence arrangements without making substantial changes to the neural network accelerator hardware and the firmware used by the accelerator” (DEISHER, paragraph [0033]).
Claims 3 and 15 are rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of FANG, and further in view of Jin H. et al in , "TurboDL: Improving the CNN Training on GPU With Fine-Grained Multi-Streaming Scheduling," published on May 4, 2020, available on
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9086163 , (hereafter, JIN).
Claim 3:
Regarding claim 3, TIMOFEJEVS in view of LEGRAND, further in view of FANG teaches the limitations in claim 1.
However, TIMOFEJEVS in view of LEGRAND, further in view of FANG did not teach the limitation “The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein work fragments may be processed out of order such that a later neural network layer may be processed before an earlier neural network layer.”
In an analogous method, JIN teaches “the method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein work fragments may be processed out of order such that a later neural network layer may be processed before an earlier neural network layer.”
See JIN in page 562, Section 4.3 Resource Evaluation, mention “At the same time, there are not periodic synchronizations between layers, and the kernels on non-critical paths mostly overlap with other kernels from different layers. This complete asynchronous execution is achieved thus TurboDL can utilize resources from different layers and balance the workloads between different layers as much as possible.”
Further, see Jin in page 561, section 4.2 Running Time Evaluation, where JIN describes "Since TurboDL is based on the critical path and the asynchronous execution of sub-tasks, it can fill the idle resources between the layers with non-critical calculations, resulting in higher performance. " Here, JIN shows running the subtasks from neural network layers in an asynchronous fashion (i.e. out of order processing).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of TIMOFEJEVS, LEGRAND, FANG and incorporate into the teachings of JIN by using the teachings of TIMOFEJEVS, LEGRAND, and FANG of dividing a neural network into subsets with JIN’s teaching of an out of order processing of the work fragments in a neural network.
One of ordinary skill in the art would be motivated to do so because by integrating JIN’s framework into the methods of TIMOFEJEVS, LEGRAND, and FANG, one with ordinary skill in the art would achieve the goal of providing a method where “multi-stream concurrency is an efficient way to utilize the characteristic of GPUs multi-cores. For the complicated GPU workloads such as CNN training, our studies suggest that the fine-grained parallelism is feasible. Through dependency analysis and data parallelism, we can decompose the task into independent sub-tasks with each sub-task being run by a kernel concurrently, so as to improve resource utilization in GPU”, (JIN, page 554, column 1, section 2.2 Data Dependency Analysis and Fine-Grained Task Partitioning), and provide “an efficient deep learning framework called TurboDL. The experimental results show that TurboDL is able to improve the performance by approximately 30 percent,” (JIN, page 553, Introduction, column 2, second to last paragraph).
Claim 15:
Regarding claim 15, TIMOFEJEVS in view of LEGRAND, further in view of FANG, teaches the limitations in claim 12. Regarding claim 15, the claim recites similar limitations as corresponding claim 3 and is rejected for similar reasons as claim 3 using similar teachings and rationale.
Claims 4 and 17 are rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of FANG, and further in view of Brothers III, J. et al. (US PG Pub. No. US 20210303992 A1), published on September 30, 2021, (hereafter, BROTHERS).
Claim 4:
Regarding claim 4, TIMOFEJEVS in view of LEGRAND, further in view of FANG, teaches the limitations in claim 1. Referring to claim 4, however, TIMOFEJEVS in view of LEGRAND, further in view of FANG, did not explicitly teach "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 further comprising: decompressing said first subset of weight matrix data loaded from said external memory."
In an analogous method, BROTHERS teaches "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 further comprising: decompressing said first subset of weight matrix data loaded from said external memory."
See BROTHERS in paragraphs [0006], describe "One common neural network layer operation, which is performed by, for example, fully connected neural network layers, that use weight values, is a matrix-vector multiply operation, in which an input vector to be processed by the neural network layer is multiplied by a weight matrix, for example to perform a multiply-accumulate operation to provide a weighted sum of the data points represented by the input vector to generate a data point of an output data array ..." BROTHERS shows this applies to multilayer neural network, where multilayer is construed to mean more than one layer.
Further, see BROTHERS in paragraph [0091] mention "In an embodiment the weight values are subdivided into respective, separate groups of data values such that each group has (substantially) the same particular, in an embodiment selected, in an embodiment predetermined, data size in their uncompressed form, and, e.g., and in an embodiment, based on the capacity (size) of any (local) storage that's used for the decompressed weight values", Further, see BROTHERS in paragraph [0070] note "Correspondingly, a (and each) set of weight values that is to be used can comprise and be in the form of any suitable and desired set of weight values, such as being in the form of an appropriate weight matrix, and/or a kernel of weight values (e.g. for use in a convolution operation (a convolution layer))."
See BROTHERS in paragraph [0141] describe "The weight values may be provided to the processor that is to perform the neural network processing (and, e.g., stored in the memory) in an uncompressed form. However, in an embodiment, the weight values are compressed before they are provided to the processor, with the processor then operating to decompress the weight values before then using them for the neural network processing. In this case, the weight values are in an embodiment re-ordered into the desired order, and then compressed, with the compressed representation of the weights then being packed so as to provide a continuous stream of compressed weights, e.g. in the memory (at a contiguous sequence of memory address), which the processor that is performing the neural network processing can then read (and decompress) in use." Here, BROTHERS notes using the method of decompressing weight values which are part of the weight matrix data that generated an output data array (i.e. first subset) that was stored in the memory before providing this to the processor.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of TIMOFEJEVS, LEGRAND, and FANG and incorporate into the teachings of BROTHERS by using the teachings of TIMOFEJEVS, LEGRAND, and FANG for a multilayer neural network model that can process weight matrix subsets with BROTHERS’ teaching of decompressing a weight matrix loaded from external memory.
One of ordinary skill in the art would be motivated to do so because by integrating BROTHERS’ framework into the methods of TIMOFEJEVS, LEGRAND, and FANG, one with ordinary skill in the art would achieve the goal of providing “Equally, any lossless compression scheme will be variable rate, and having the weights reordered and fully packed together makes such schemes simpler and more efficient (as weights compress better with full packing).” (BROTHERS, paragraph [0148]).
Claim 17:
Regarding claim 17, TIMOFEJEVS in view of LEGRAND, and further in view of FANG teaches the limitations in claim 12. Regarding claim 17, the claim recites similar limitations as corresponding claim 4 and is rejected for similar reasons as claim 4 using similar teachings and rationale.
Claim 5 is rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of FANG, further in view of Kumar, R. et al., (US PG Pub. No. US 20190303750 A1), published on October 3, 2019, (hereafter, KUMAR).
Claim 5:
Regarding claim 5, TIMOFEJEVS in view of LEGRAND, further in view of FANG, teach the limitations in claim 1.
Referring to claim 5, however, TIMOFEJEVS in view of LEGRAND, further in view of FANG, did not explicitly teach “The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein said first subset of weight matrix data may comprise one of several different data precisions.”
In an analogous method, KUMAR teaches “The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 wherein said first subset of weight matrix data may comprise one of several different data precisions,”
See KUMAR shows in paragraph [0024] that "FIG. 3 depicts an example of hashing based weight sharing scheme for DNNs. Various embodiments provide a reconfigurable weight sharing scheme employing hash functions to generate coordinates and values of virtual weights in a matrix. A synaptic weight is grouped into a random set of connections, such that the connections in the same hash bucket share the same weight. A real weight matrix contains the unique weight values and is stored on-chip with the computation engine that performs computations using inputs and weight values. For example, a real weight value can be of any integer (8, 16-bit) or any floating-point precision (half, single or double) number depending on the application and hardware." Here, KUMAR shows the weight matrix data values can take any integer, or decimal quantity (such as from 8 or 16 bit) floating point precisions, which teaches one of several precisions. See the instant application’s specification paragraph [0048] for more information on data precision levels.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine TIMOFEJEVS, LEGRAND, and FANG, with the teachings of KUMAR by using the teachings of TIMOFEJEVS, LEGRAND, and FANG of using a multilayer neural network to process subsets for weight matrix data, with KUMAR’s teaching of the processing the subset of weight matrix data can take on different data precisions.
One of ordinary skill in the art would be motivated to do so because by integrating KUMAR’s framework into the methods of TIMOFEJEVS, LEGRAND, and FANG, one with ordinary skill in the art would achieve the goal of providing “the reconstruction of the weight matrix leverages compute than memory, thereby providing high energy-efficiency due to very limited amount of memory accesses,” (KUMAR, paragraph [0019]).
Claim 6 is rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of FANG, further in view of Estep, P. et al., (US PG Pub. US 20230068168 A1), published March 2, 2023, (hereafter, ESTEP).
Claim 6:
Regarding claim 6, TIMOFEJEVS in view of LEGRAND, further in view of FANG, teaches the limitations in claim 1.
However, TIMOFEJEVS in view of LEGRAND, further in view of FANG, did not explicitly teach "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 further comprising:
reloading in said first subset of weight matrix data from said external memory after a context switch of said neural network processor."
In an analogous method, ESTEP teaches "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 1 further comprising: reloading in said first subset of weight matrix data from said external memory after a context switch of said neural network processor,"
See ESTEP in paragraph [0095] describe "In an example, the PAU core 606 is a pipelined processor such that multiple stages of different instructions are executed together per clock cycle. The PAU core 606 can include a barrel-multithreaded processor, with thread control 604 circuitry to switch between different register files (e.g., sets of registers containing current processing state) upon each clock cycle. This enables efficient context switching between currently executing threads. " Here, ESTEP introduces the phrase context switching. In addition, ESTEP mentions that "switch between different register files" shows a context switch of the processor of the neural network.
See ESTEP in paragraph [0038], where ESTEP shows "in an example, changing the fully connected layer values to correspond to values in the matrix prior to the transposition includes mapping a first position of the matrix to a second position in the fully connected layer under a flatten operation of the neural network model and moving a weight, of the fully connected layer, that corresponds to the first position to the second position. The flatten operation transforms a matrix into an array. For example, the flatten operation illustrated in FIG. 2 starts at the upper left and proceeds to the right down a row until the row end is met, placing each encountered value into the array of the FC layer. The operation then continues with the next row until all the values of the source matrix (e.g., matrix 222 or matrix 220) are placed in the array (e.g., array 224 or array 232)...When data structures other than a matrix are used, a similar updating of values, or the movement of objects, fields, etc., may be used to effectuate the movement of connections to correspond with the movement of neurons." Note the examiner construes reloading to mean similar definition as updating and then loading the updated information. Here, ESTEP teaches updating the values for matrix data and non-matrix data, until all values of matrix are filled (i.e. reloaded), and refers to a first position of the matrix, (i.e. first subset of weight matrix data).
Further, see ESTEP in paragraph [0030] mention "FIG. 1 illustrates example transitions between processing circuitry for a neural network, according to an embodiment. As illustrated, a host processor 102 runs a neural network model 106 on an accelerator 104 (e.g., a SIMD processor) to eventually produce an output by the neural network model 106, such as image classification, feature detection, etc. The interim operation 108 illustrates a transition of processing during the run from the accelerator 104 back to the host processor 102, or an equivalent processor, to provide some computation before the run can continue on the accelerator 104, such as implementing a transpose layer in a CNN as described above." Here, ESTEP elaborates what the transition of processing (i.e. context switch) can be, such as to image classification, feature detection. See ESTEP in paragraphs [0036], [0055], [0089], and [0115] for more information.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of TIMOFEJEVS, LEGRAND, and FANG and incorporate into the teachings of ESTEP by using the teachings of TIMOFEJEVS, LEGRAND, and FANG for a multilayer neural network model that can process weight matrix subsets with ESTEP’s teaching of reloading in a weight matrix data from said external memory after a context switch of the neural network processor.
One of ordinary skill in the art would be motivated to do so because by integrating ESTEP’s framework into the methods of TIMOFEJEVS, LEGRAND, and FANG, one with ordinary skill in the art would achieve the goal of providing " a good balance in distributed processing resources to efficiently implement neural networks that is made more efficient by the neural network transpose layer removal described herein. When combined the CNM system 402 enables high performance implementations of practical neural network solutions to many problems" (ESTEP, paragraph [0062]).
Claims 7 and 8 are rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of Jain N. et al., (US PG Pub. No. US20220012592 A1), published on January 13, 2022, (hereafter, JAIN).
Claim 7:
Regarding claim 7, TIMOFEJEVS teaches “A method of processing a multilayer neural network with a neural network processor and tapering out weight matrix from said neural network processor, said method comprising the steps of: dividing said multilayer neural network into subsets of neural network layers wherein each subset will be processed as a group; said subsets of neural network layers referred to as partition;”
See TIMOFEJEVS in paragraph [0016] describe "In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network." Note, the examiner construes the word partition to mean any subdivision of a neural network model. Here, TIMOFEJEVS describes an implementation that sub-divides a neural network first from layers to sub-layers, then from sub-layers into sub-function blocks. TIMOFEJEVS in paragraph [0016] mentioned “Each sub-layer implements an intermediate mathematical function”, relates to (i.e. each subset will be processed as a group ). Regarding the subset of neural network layers referred to as a partition limitation, the instant application’s specification paragraph [0085] states “each partition of neural network layers may be processed together as a group.”
Further, see TIMOFEJEVS in paragraph [0145] “2. If K>N then: a. Divide K input neurons into m1 =
PNG
media_image1.png
103
75
media_image1.png
Greyscale
groups such that every group consists of no more than N inputs. b. Construct the first hidden layer LTHi of the T-NN from rr^neurons, each neuron performing an identity activation function. c. Connect input neurons from every group to corresponding neuron from the next layer.” Here, TIMOFEJEVS describes a part of the neural network that handles data is treated as a group or subdivision (i.e. each said subset of neural network layers referred to as a partition) See TIMOFEJEVS for more information in paragraphs [0142-0143] and [00282].
Further, TIMOFEJEVS teaches “dividing each network layer in each partition into a set of work fragments;”
See TIMOFEJEVS in paragraph [0016] describe "in some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network." Here, TIMOFEJEVS shows the sub-function blocks as the work fragments from “selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer” (i.e. dividing each neural network layer of each said partition into a set of work fragments). This shows that each sub-function blocks have its own math function and calculations associated with its respective task, and further indicates the segmented nature of a neural network.
However, TIMOFEJEVS did not teach
“grouping set of said work fragments of each cut into work fragment subsets that can be processed simultaneously;”
“loading into said neural network processor a first work fragment subset for a first partition from said external memory;”
“loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition;”
“commencing processing of said work fragments for said neural network layers of said first partition;
discarding said first weight matrix for a first neural network fragment of said first partition after processing a final work fragment for said first network layer to free memory resources;”
“and loading a second weight matrix for a neural network layer in a subsequent partition into said neural network processor while completing processing of said work fragments of said first partition.”
In an analogous method, LEGRAND teaches “grouping set of said work fragments of each cut into work fragment subsets that can be processed simultaneously;”
See LEGRAND in paragraph [0020] describe “as shown in FIG. 1, however, the weight matrix 120 may be split among a plurality of different computer processors, and the processors may generate different portions of the matrix 130 in parallel. For example, the weight matrix 120 may be striped row-wise (separated into subsets of rows), and each processor may be provided with a different subset of the rows. The input matrix 110 may be striped column-wise (separated into subsets of columns), and each processor may be provided with a different subset of the columns. … In some embodiments, the matrix 130 may be generated by performing a series of "reduction" operations or some equivalent operation in which multiple sets of numbers— the intermediate matrices in this example— are reduced into a single set of numbers— the subset of columns of matrix 130 to be stored on an individual processor. A reduction operation can be performed to aggregate, from the intermediate matrices, each separate subset of columns to be stored on each individual processor. In some cases, the reduction operations may be performed substantially in parallel or otherwise at least partially overlapping in time.” Here, LEGRAND shows that each processor has different portions of data named matrix 130. Then, LEGRAND describes that grouping different portions of the matrix data (i.e. relate to partition) with reduction operations being a work fragment, that may be performed substantially in parallel or otherwise at least partially overlapping in time (i.e. work fragment of each partition processed simultaneously). Note the examiner construes work fragment to mean any task, job, part of a data, that is performed on data.
Further, see LEGRAND in paragraph [0004] for more details “FIG. 1 is a diagram of an illustrative artificial neural network with multiple layers, indicating how the layers are to be distributed among multiple computer processors for parallel processing.” LEGRAND further elaborates that the process described in [0020] is part of neural networks with multiple layers, and each layer needs a number of processors to process data in parallel or simultaneously.
Further, LEGRAND teaches “loading into said neural network processor a first work fragment subset for a first partition from said external memory;”
See LEGRAND in paragraph [0028] describe “..the individual computer processors multiply their own subsets of columns of the current matrix by their own subsets of rows of the current weight matrix. In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors.” LEGRAND mentions subsets of data are loaded onto corresponding processors. See LEGRAND in paragraph [0023] for more details on loading from memory.
Further, see LEGRAND describe in [0002] “Sets of individual input vectors ("mini-batches") may be processed at the same time by using an input matrix instead of a single input vector. The NN can repeatedly process the input data, and the parameters (e.g., the weight matrices) of the NN can be modified in what amounts to a trial-and-error process until the model produces (or "converges" on) the correct or preferred output.” LEGRAND describes this can be done repeatedly for input data of the neural network, … and includes loading data for any work fragment for any layer or partition of the neural network.
Later in paragraph [0054] Clause 1. LEGRAND describes “a system comprising a plurality of processors, the system programmed by executable instructions to at least: obtain data defining an artificial neural network, the artificial neural network comprising a first layer of nodes, a second layer of nodes, and a third layer of nodes, wherein the first layer comprises more nodes than the second layer, and wherein the third layer comprises more nodes than the second layer; provide to a first processor of the plurality of processors: a first column of input data from a first data matrix, the first data matrix comprising input data for the artificial neural network; a first row of weights from a first weight matrix, the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer; and a first column of weights from a second weight matrix, the second weight matrix comprising weights for connections between nodes of the second layer and nodes of the third layer; provide to a second processor of the plurality of processors: a second column of input data from the first data matrix; a second row of weights from the first weight matrix; and a second column of weights from the second weight matrix; compute, using the first processor, a first subset of columns of a second data matrix of values for the second layer, wherein the first subset is computed from the first column of input data, the first row of weights, and aggregated values received from the second processor of the plurality of processors”. Note the examiner construes loading data to mean similar to obtaining data. Here, LEGRAND mentions loading into a processor of the neural network and specifies the obtaining or loading of the column of weights (i.e. first work fragment) for a first layer of the first subset (i.e. first partition) of a neural network.
Further, LEGRAND teaches “loading in a first weight matrix from said external memory for said set of work fragments layers of said first partition;”
See LEGRAND in paragraph [0043] describe “FIG. 5 and otherwise described herein, a computing system 500 may include various other components, such as one or more network interfaces (e.g., network interface cards), one or more computer readable medium drives (e.g., high density disks, solid state drives, flash drives, and/or other persistent non-transitory computer-readable media), an input/output device interface (e.g. an 10 interface in communication with one or more microphones or display screens), and one or more computer readable memories (e.g., random access memory and/or other volatile non-transitory computer-readable media).” Note the examiner construes external memory to mean any storage device used for long-term data storage, such as flash drives. Here, LEGRAND describes devices used for long-term storage for external memory.
Later, see LEGRAND in paragraph [0054] Clause 1. describe “a system comprising a plurality of processors, the system programmed by executable instructions to at least: obtain data defining an artificial neural network, the artificial neural network comprising a first layer of nodes, a second layer of nodes, and a third layer of nodes, wherein the first layer comprises more nodes than the second layer, … provide to a first processor of the plurality of processors: a first column of input data from a first data matrix, the first data matrix comprising input data for the artificial neural network; a first row of weights from a first weight matrix, the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer; and a first column of weights from a second weight matrix, the second weight matrix comprising weights for connections between nodes of the second layer and nodes of the third layer; provide to a second processor of the plurality of processors: a second column of input data from the first data matrix; a second row of weights from the first weight matrix; and a second column of weights from the second weight matrix; compute, using the first processor, a first subset of columns of a second data matrix of values for the second layer, wherein the first subset is computed from the first column of input data, the first row of weights, and aggregated values received from the second processor of the plurality of processors”. Here, when LEGRAND mentions obtaining “the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer” into a processor within a neural network, this shows loading in a first subset of weight matrix data from said external memory for work fragments of a first partition into said neural network processor, where the weights for connections between nodes of first and second layers are the first subset. The first partition in this case is any part of the first layer of the neural network that LEGRAND shows.
Further, LEGRAND teaches “commencing processing of said work fragments for said neural network layers of said first partition;”
See LEGRAND in paragraph [0028] describe “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors. This may happen before processing input matrices because the structure of the NN 100 (the sizes of the respective layers, and therefore the sizes of the input and output matrices for each NN operation) is predetermined”. Here, LEGRAND mentions the processing of any part of the input matrices (i.e. work fragment subset) starts when the weight matrices are made accessible (i.e. available) to the corresponding computer processors. Note the examiner construes available to mean accessible or in a ready to process format. See LEGRAND in paragraph [0036] for more details.
Further, LEGRAND teaches “and loading a second weight matrix for a neural network layer in a subsequent partition into said neural network processor while completing processing of said work fragments of said first partition.”
See LEGRAND in paragraph [0028] mention “ In some embodiments, the subsets of rows of the current weight matrix 120 have already been stored on, or are otherwise accessible by, the corresponding individual computer processors. For example, when the NN 100 is loaded on the computing system 500, when the process 200 is initiated, or at some other time, the subsets of rows and/or columns of the various weight matrices of the NN 100 may be stored on or otherwise made accessible to the corresponding computer processors.” Here, in [0028], LEGRAND shows the relevant subsets and work fragments of the neural network have been loaded onto the processor from memory.
Further, see LEGRAND in paragraph [0054] Clause 5, describe “computing, by the plurality of computer processors, the second layer matrix, wherein individual computer processors of the plurality of computer processors each compute a different contribution to the second layer matrix using a corresponding subset of columns of the first layer matrix and a corresponding subset of rows of a first weight matrix, and wherein the first weight matrix comprises weights for connections between nodes of the first layer and nodes of the second layer;” Here, LEGRAND shows the second layer matrix is part of the second work fragment subset and is used to compute information that correspond to a subset of columns from the first layer matrix, (i.e. loading second work fragment subset for said first partition). See LEGRAND for more information in paragraphs [0011] and [0018].
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of TIMOFEJEVS and incorporate into the teachings of LEGRAND because both references teach processing a neural network into partitions and subsets.
One of ordinary skill in the art would be motivated to do so because these methods “provide efficient parallelization for artificial neural network processing by minimizing or otherwise reducing the communication required between individual computer processors for performing various artificial neural network operations in parallel. Such efficiently parallelized artificial neural networks may be used in a variety of machine learning applications and other systems, including but not limited to: product recommendation generation, automatic speech recognition, facial recognition, handwriting recognition, and image recognition” (LEGRAND, [0010]).
However, TIMOFEJEVS in view of LEGRAND did not teach “discarding said first weight matrix for a first neural network fragment of said first partition after processing a final work fragment for said first network layer to free memory resources.”
In an analogous method, JAIN teaches “discarding said first weight matrix for a first neural network fragment of said first partition after processing a final work fragment for said first network layer to free memory resources.”
See JAIN in paragraph [0025] mention “small batch size artificial intelligence utilizes significant memory bandwidth, such as when an artificial intelligence system fetches neural network weights (e.g., a weight matrix) from a dynamic random-access memory (DRAM) to perform an inference. As such, the weight matrix can cause bottlenecking that limits a rate at which the neural network performs an inference.” JAIN provides context that the weights are part of weight matrix data in neural networks, and that these weights are part of a memory. Here, JAIN addresses an issue of bottlenecking which can consumes significant memory bandwidth. See JAIN in paragraph [0026] for details.
Later, JAIN notes in paragraph [0044] that “in FIG. 2, the memory 212 organizes the meta-data and the compressed data by cache lines. For example, a first cache line of the memory 212 can include the meta-data for each of the respective weight matrices. Additionally, the memory 212 can include the compressed data associated with a first weight matrix in a first set of cache lines positioned after the first cache line.” JAIN describes that the memory that includes weight matrix data can be associated with a first weight matrix information which contains information regarding a first neural network fragment of a first partition.
Further, see JAIN in paragraph [0103] describe “At block 1104, the matrix compressing circuitry 104 prunes the data (e.g., a tile of neural network weights). For example, the pruning circuitry 204 (FIG. 2) can prune the data. Specifically, the pruning circuitry 204 converts the uncompressed data to partially compressed data by removing weights below a certain threshold.” Here, JAIN specifically mentions removing weights that are part of a weight matrix. Removing is interpreted to mean the same as discarding. During this process, the matrix compressing circuitry 104 and pruning circuitry 204 processes the data as described in [0103]. Since weights are part of the weight matrix data, this can be data from a first weight matrix that is removed.
Further, see JAIN in paragraph [0061] describe “In FIG. 4, the pruned data 404 corresponds to a first tile of neural network weights utilized by the neural network circuitry 102. In some examples, the neural network circuitry 102 utilizes sixty four tiles of neural network weights to perform an inference. Accordingly, the matrix compressing circuitry 104 may compress a remaining sixty three tiles similar to the first tile. Similarly, the matrix compressing circuitry 104 can generate a respective byte of meta-data, similar to the meta-data 402, for the respective remaining sixty three tiles.” JAIN here mentions that the matrix compressing circuitry 104 also plays a role in processing remaining tiles of neural network weight values (i.e. somewhere in the remaining tiles includes processing a final work fragment) that is part of the inference task of the first layer of the neural network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of TIMOFEJEVS and LEGRAND, and incorporate into the teachings of JAIN by using the teachings of TIMOFEJEVS and LEGRAND, for a multilayer neural network model that can process weight matrix subsets with JAIN’s teaching of discarding a first weight matrix for a first neural network fragment of a first partition after processing the final work fragment to free memory resources.
One of ordinary skill in the art would be motivated to do so because by integrating JAIN’s framework into the methods of TIMOFEJEVS and LEGRAND, one with ordinary skill in the art would achieve a goal of providing a method to “improve compute efficiency (e.g., tile matrix multiplying via AMX, execution units on GPUs) in addition to utilizing bandwidth efficiently. Specifically, the example program can improve data efficiency of a cache such that the cache improves from a low-level cache (LLC) to a mid-level cache (MLC),” (JAIN, paragraph [0030]).
Claim 8:
Regarding claim 8, TIMOFEJEVS, in view of LEGRAND, and further in view of JAIN teaches the limitations in claim 7.
Further, JAIN teaches "The method of processing a multilayer neural network with a neural network processor and tapering weight matrix data from said neural network processor as set forth in claim 7, said method further comprising: decompressing said first weight matrix loaded from said external memory."
See JAIN in paragraph [0158] describe “example methods, apparatus and articles of manufacture have been disclosed that accelerate compression and/or decompression of quantized neural networks utilizing unstructured sparsity. The examples disclosed herein reduce a memory bandwidth requirement and improves compute efficiency to enable accelerated inferences in neural networks. The disclosed methods, apparatus and articles of manufacture improve the efficiency of using a computing device reducing a memory bandwidth utilized to decompress data (e.g., neural network weights)”. Here, JAIN describes using the method of decompression on neural networks. JAIN further describes in abstract “Methods, apparatus, systems, and articles of manufacture to perform weight and activation compression and decompression are disclosed. An example apparatus includes memory, instructions in the apparatus, and processor circuitry to execute the instructions to execute a compression operation to obtain compressed data corresponding to weights in a weight matrix, and determine meta-data associated with the weight matrix, …”
Further, see JAIN in paragraph [0118], elaborating “FIG. 13 is a flowchart representative of example machine readable instructions and/or example operations 1300 that may be executed and/or instantiated by processor circuitry to implement the matrix decompressing circuitry 106 of FIGS. 1, 6, 7, 8, and/or 10 to decompress data corresponding to tiles of neural network weights. The machine readable instructions and/or operations 1300 of FIG. 13 begin at block 1302, at which the matrix decompressing circuitry 106 identifies a tile to load. For example, the data transceiver 602 (FIG. 6) can receive a signal indicative of the tile from the neural network circuitry 102 (FIG. 1) via the bus 108 (FIGS. 1, 2, and 6). In some examples, the data transceiver 602 receives an address of the tile indicative of a location (e.g., an address) of the tile respective to other tiles in an associated tile array.” Here, JAIN refer to neural network weights as part of weight values in a matrix. See JAIN in paragraph [0075] notes “FIG. 7 is a block diagram of an example implementation of the matrix decompressing circuitry 106 in an example matrix (e.g., tile) operating system 700”. Here, JAIN shows that the matrix decompressing circuitry 106 acts on matrix data which includes weight matrix values mentioned in [0118]. Note the examiner construes first to mean initial or earliest. By identifying a tile to load, this shows initial or first loading of a weight matrix into memory. See JAIN in the abstract and paragraphs [0073, 0077-0080, 0122-0123] for more information.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of TIMOFEJEVS and LEGRAND, and incorporate into the teachings of JAIN by using the teachings of TIMOFEJEVS and LEGRAND, for a multilayer neural network model that can process weight matrix subsets with JAIN’s teaching of decompressing a weight matrix loaded from external memory.
One of ordinary skill in the art would be motivated to do so because by integrating JAIN’s framework into the methods of TIMOFEJEVS and LEGRAND, one with ordinary skill in the art would achieve a goal of providing a method to “improve compute efficiency (e.g., tile matrix multiplying via AMX, execution units on GPUs) in addition to utilizing bandwidth efficiently. Specifically, the example program can improve data efficiency of a cache such that the cache improves from a low-level cache (LLC) to a mid-level cache (MLC),” (JAIN, paragraph [0030]).
Claim 9 is rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of JAIN, and further in view of KUMAR.
Claim 9:
Regarding claim 9, TIMOFEJEVS in view of LEGRAND, further in view of JAIN, teach the limitations in claim 7. Referring to claim 9, however, TIMOFEJEVS in view of LEGRAND, further in view of JAIN, did not explicitly teach “the method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 wherein said first weight matrix may comprise one of several different data precisions.”
In an analogous method, KUMAR teaches “the method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 wherein said first weight matrix may comprise one of several different data precisions,”
See KUMAR shows in paragraph [0024] that "FIG. 3 depicts an example of hashing based weight sharing scheme for DNNs. Various embodiments provide a reconfigurable weight sharing scheme employing hash functions to generate coordinates and values of virtual weights in a matrix. A synaptic weight is grouped into a random set of connections, such that the connections in the same hash bucket share the same weight. A real weight matrix contains the unique weight values and is stored on-chip with the computation engine that performs computations using inputs and weight values. For example, a real weight value can be of any integer (8, 16-bit) or any floating-point precision (half, single or double) number depending on the application and hardware." Here, KUMAR shows the weight matrix data values can take any integer, or decimal quantity (such as from 8 or 16 bit) floating point precisions, which teaches one of several precisions. See the instant application’s specification paragraph [0048] for more information on data precision levels.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine TIMOFEJEVS, LEGRAND, and JAIN, with the teachings of KUMAR by using the teachings of TIMOFEJEVS, LEGRAND, and JAIN of using a multilayer neural network to process subsets for weight matrix data, with KUMAR’s teaching of the processing the subset of weight matrix data can take on different data precisions.
One of ordinary skill in the art would be motivated to do so because by integrating KUMAR’s framework into the methods of TIMOFEJEVS, LEGRAND, and JAIN, one with ordinary skill in the art would achieve the goal of providing “the reconstruction of the weight matrix leverages compute than memory, thereby providing high energy-efficiency due to very limited amount of memory accesses,” (KUMAR, paragraph [0019]).
Claim 10 is rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of JAIN, further in view of ESTEP.
Claim 10:
Regarding claim 10, TIMOFEJEVS in view of LEGRAND, further in view of JAIN, teaches the limitations in claim 7.
However, TIMOFEJEVS in view of LEGRAND, further in view of JAIN, did not explicitly teach "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 further comprising: reloading in said first weight matrix from said external memory after a context switch of said neural network processor."
In an analogous method, ESTEP teaches "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 further comprising: reloading in said first weight matrix from said external memory after a context switch of said neural network processor,"
See ESTEP in paragraph [0095] describe "In an example, the PAU core 606 is a pipelined processor such that multiple stages of different instructions are executed together per clock cycle. The PAU core 606 can include a barrel-multithreaded processor, with thread control 604 circuitry to switch between different register files (e.g., sets of registers containing current processing state) upon each clock cycle. This enables efficient context switching between currently executing threads. Here, ESTEP introduces the phrase context switching. In addition, ESTEP mentions that "switch between different register files" shows a context switch of the processor of the neural network.
See ESTEP in paragraph [0038], where ESTEP shows "changing the fully connected layer values to correspond to values in the matrix prior to the transposition includes mapping a first position of the matrix to a second position in the fully connected layer under a flatten operation of the neural network model and moving a weight, of the fully connected layer, that corresponds to the first position to the second position. The flatten operation transforms a matrix into an array. For example, the flatten operation illustrated in FIG. 2 starts at the upper left and proceeds to the right down a row until the row end is met, placing each encountered value into the array of the FC layer. The operation then continues with the next row until all the values of the source matrix (e.g., matrix 222 or matrix 220) are placed in the array (e.g., array 224 or array 232)...When data structures other than a matrix are used, a similar updating of values, or the movement of objects, fields, etc., may be used to effectuate the movement of connections to correspond with the movement of neurons." Note the examiner construes reloading to mean similar definition as updating and then loading the updated information. Here, ESTEP teaches updating the values for matrix data and non-matrix data, until all values of matrix are filled (i.e. reloaded), and refers to a first position of the matrix, (i.e. first subset of weight matrix data).
Further, see ESTEP in paragraph [0030] mention "FIG. 1 illustrates example transitions between processing circuitry for a neural network, according to an embodiment. As illustrated, a host processor 102 runs a neural network model 106 on an accelerator 104 (e.g., a SIMD processor) to eventually produce an output by the neural network model 106, such as image classification, feature detection, etc. The interim operation 108 illustrates a transition of processing during the run from the accelerator 104 back to the host processor 102, or an equivalent processor, to provide some computation before the run can continue on the accelerator 104, such as implementing a transpose layer in a CNN as described above." Here, ESTEP elaborates what the transition of processing (i.e. context switch) can be, such as to image classification, feature detection. See ESTEP in paragraphs [0036], [0055], [0089], and [0115] for more information.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of TIMOFEJEVS, LEGRAND, and JAIN and incorporate into the teachings of ESTEP by using the teachings of TIMOFEJEVS, LEGRAND, and JAIN for a multilayer neural network model that can process weight matrix subsets with ESTEP’s teaching of reloading in a weight matrix data from said external memory after a context switch of the neural network processor.
One of ordinary skill in the art would be motivated to do so because by integrating ESTEP’s framework into the methods of TIMOFEJEVS, LEGRAND, and JAIN, one with ordinary skill in the art would achieve the goal of providing " a good balance in distributed processing resources to efficiently implement neural networks that is made more efficient by the neural network transpose layer removal described herein. When combined the CNM system 402 enables high performance implementations of practical neural network solutions to many problems" (ESTEP, paragraph [0062]).
Claim 11 is rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of JAIN, further in view of Saguil D. et al., in "A Layer-Partitioning Approach for Faster Execution of Neural Network-Based Embedded Applications in Edge Networks," published on March 17, 2020, available in https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9039657, (hereafter, SAGUIL).
Claim 11:
Regarding claim 11, TIMOFEJEVS in view of LEGRAND, further in view of JAIN, teaches the limitations in claim 7.
However, TIMOFEJEVS in view of LEGRAND, further in view of JAIN, did not teach "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 wherein said partitions can belong to different neural networks."
In an analogous method, SAGUIL teaches "The method of processing a multilayer neural network with a neural network processor and managing access to a memory as set forth in claim 7 wherein said partitions can belong to different neural networks."
See SAGUIL in paragraph 7, Introduction section describe "To find these possible improvements, a simulated IoT environment was prepared along with several NN models. We then executed each NN models using several datasets to measure the execution time of each layer on both an embedded system and a fog node. From this, we determined if the model should be offloaded to the fog node at all (i.e. the embedded system runs the model at a rate faster than the network latency added to the fog node execution time). After measuring the execution time of each layer on both devices, we then plotted splitting points in each of the models and implemented them in the following experiment phase." Here, SAGUIL mentions using the method or technique where the partitions can be applied to different neural network models.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine TIMOFEJEVS, LEGRAND, and JAIN, with the teachings of SAGUIL by using the teachings of TIMOFEJEVS, LEGRAND, and JAIN of using a multilayer neural network to process subsets for weight matrix data, with SAGUIL’s teaching of the partitions belong to different neural networks.
One of ordinary skill in the art would be motivated to do so because by integrating SAGUIL’s framework into the methods of TIMOFEJEVS, LEGRAND, and JAIN, one with ordinary skill in the art would achieve the goal of providing a method so “ the IoT device only needs to send messages only when they were not confident in their result, rather than every time, thus increasing time efficiency,” (SAGUIL, page 59457, col. 2, paragraph before bulleted list).
Claim 13 is rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of FANG, further in view of Shi, Z. et al., in “A hierarchical neural model of data prefetching”, published for a conference between April 19-23, 2021, available on https://dl.acm.org/doi/pdf/10.1145/3445814.3446752, (hereafter, SHI).
Claim 13:
Regarding claim 13, TIMOFEJEVS in view of LEGRAND, further in view of FANG teaches the limitations of claim 12.
However, referring to claim 13, TIMOFEJEVS in view of LEGRAND, further in view of FANG, did not teach “The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said prefetching is performed with a lower priority than other accesses to said external memory.”
In an analogous method, SHI teaches “The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said prefetching is performed with a lower priority than other accesses to said external memory,”
See SHI in page 862, col. 2, section I. Introduction, first bullet point mention “we outline the design space of temporal prefetchers by using the notion of features and localization, and we show that neural networks are capable of exploiting rich features, such as the history of data addresses.” See section I. Introduction, paragraph 2, in page 861 for more details.
Further, SHI in page 868, first full paragraph, section 5.1 Methodology, after Table 3., describes “all prefetchers are situated at the last-level cache (LLC), which means that their inputs are LLC accesses”. Here, SHI talks about prefetching as a last level cache or shows prefetching to be performed as one of the last processing tasks in the cache than other tasks in memory. See Table 3 for more details.
PNG
media_image2.png
240
510
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of TIMOFEJEVS, LEGRAND, and FANG and incorporate into the teachings of SHI by using the teachings of TIMOFEJEVS, LEGRAND, and FANG for a multilayer neural network model that can process weight matrix subsets with SHI’s teaching of prefetching performed with a lower priority than other accesses to said external memory.
One of ordinary skill in the art would be motivated to do so because by integrating SHI’s framework into the methods of TIMOFEJEVS, LEGRAND, and FANG, one with ordinary skill in the art would achieve the goal of providing a method “compared to the Delta-LSTM prefetcher [13], Voyager’s hierarchical representation yields significant storage and computational efficiency. In particular, Voyager reduces the training overhead by 15.1× and prediction latency by 15.6×.” (SHI, page 870, section 5.4 Model Compression and Overhead, first paragraph).
Claims 14 and 16 are rejected under 35 U.S.C. 103 over TIMOFEJEVS in view of LEGRAND, further in view of FANG, and further in view of SAGUIL.
Claim 14:
Regarding claim 14, TIMOFEJEVS in view of LEGRAND, further in view of FANG, teach the limitations in claim 12.
Referring to claim 14, however, TIMOFEJEVS in view of LEGRAND, further in view of FANG, did not explicitly teach “The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said partitions can belong to different neural networks.”
In an analogous method, SAGUIL teaches “The method of processing a multilayer neural network with a neural network processor and prefetching weight matrix data from an external memory as set forth in claim 12 wherein said partitions can belong to different neural networks,”
See SAGUIL in Introduction section, page 59457, describe “See SAGUIL in page 59457, in the Introduction section, second to last paragraph, second point in the bulleted list describe " Analyzed several datasets on several different NN models to determine which of their characteristics affected the runtime on embedded systems the most." Later, see SAGUIL in page 59457, third paragraph of the introduction section, first column for more information. Here, SAGUIL mentions using the method where the partitions can be applied to different neural network models.
See page 59465 Section V. Results and Analysis, part A. Delegation Phase, last paragraph where SAGUIL describes “therefore, this model was partitioned into 3 sub-models, whereas the first executes on the embedded system and the last two execute on the fog node.” Here, SAGUIL indicates the model is partitioned.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine TIMOFEJEVS, LEGRAND, and FANG, with the teachings of SAGUIL by using the teachings of TIMOFEJEVS, LEGRAND, and FANG of using a multilayer neural network to process subsets for weight matrix data, with SAGUIL’s teaching of the partitions belong to different neural networks.
One of ordinary skill in the art would be motivated to do so because by integrating SAGUIL’s framework into the methods of TIMOFEJEVS, LEGRAND, and FANG, one with ordinary skill in the art would achieve the goal of providing a method so “ the IoT device only needs to send messages only when they were not confident in their result, rather than every time, thus increasing time efficiency,” (SAGUIL, page 59457, col. 2, paragraph before bulleted list).
Claim 16:
Regarding claim 16, TIMOFEJEVS in view of LEGRAND, further in view of FANG, teach the limitations in claim 12. Referring to claim 16, the claim recites similar limitations as corresponding claim 14 and is rejected for similar reasons as claim 14 using similar teachings and rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WENWEI ZENG whose telephone number is (571)272-7111. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WenWei Zeng/Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights