DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1, 3-8, 10-11, and 13-14 are pending.
Claim Objections
Claims 1 and 3-8 objected to because of the following informalities.
In claim 1, applicant recites “wherein the binary mask for each task is learned to selectively enable or disable columns for that task while keeping backbone model fixed,” which should read: “wherein the binary mask for each task is learned to selectively enable or disable columns for that task while keeping the backbone model fixed.”
The dependent claims are objected to by virtue of their dependency. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3, 6, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Hoang (US 20210110235; hereinafter Hoang ‘235) in view of Santoro (US 20170228637) and “Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights” by Mallya et al. (NPL previously provided by applicant; hereinafter Mallya).
Claim 1
Hoang ‘235 discloses:
A neural network accelerator architecture for multiple task adaptation {[0098] The embodiments presented above provide storage class memory array, or sub-array, for in-memory computing architectures to accelerate convolution neural network inference, i.e., tasks.}, comprising:
a random-access memory comprising a plurality of subarrays, each subarray comprising M rows and N columns of memory cells, wherein each column in the subarray is configured to store a convolution kernel {[0063] FIG. 14 is an embodiment for an architecture that can leverage all-zero columns of a storage class memory sub-array to reduce the number of bit line accesses to improve performance and energy efficiency. FIG. 14 illustrates an array, or portion of an array, 1401 of resistive non-volatile memory cells and peripheral elements, similar to the portions of the array shown in FIG. 9, but with the memory cells represented as blocks at the intersection of word lines and bit lines. N bit lines, running from BL.sup.0 to BL.sup.N−1, and M word lines, running from WL.sup.0 to WL.sup.M−1, are shown and can represent the whole of an array or a compact portion of a larger array, i.e., a plurality of subarrays, each subarray comprising M rows and N columns of memory cells, wherein each column in the subarray is configured to store a convolution kernel. random-access memory described in [0050]: For example, on a host processor executing the neural network, the weight could be read out of an SSD in which they are stored and loaded into RAM on the host device.};
a source line driver connected to a plurality of N source lines, each source line corresponding to a column in the subarray {[0067] The use of the ZCI values can provide energy savings as, when ZCI=0, there is no need to access the corresponding bit line. This is illustrated schematically with multiplex circuit MUX 1411 that receives the bit line addresses and also the ZCI values from the ZCI register 1420. If the selected bit line address matches a bit line with ZCI=0, the MUX 1411 can notify the bit line activation circuit, along with the ADC 1407, i.e., a source line driver, and shift and add 1409 so that the corresponding column can just be skipped in the sensing operation.};
a binary mask buffer memory having size at least N bits, each bit corresponding to a column in the subarray, where a 0 corresponds to turning off the column for a convolution operation and a 1 corresponds to turning on the column for the convolution operation {[0065] For this purpose, a Zero Column Index (ZCI) 1420 is added at the array or sub-array level, adding one vector per array or sub-array. The size of the ZCI 1420 is the number of bit lines in the array or subarray 1401, which can be the same as the corresponding row buffer. Each bit line BL.sup.i has an entry in ZCI 1420 where, in this embodiment, ZCI.sup.i=‘0’ indicates the i.sup.th column having all-zero weights and ZCI.sup.i=‘1’ indicates the i.sup.th column having at least one non-zero weight, i.e., a binary mask buffer.}; and
a controller configured to selectively drive each of the N source lines with a corresponding value from the mask buffer to perform task-specific inference {[0067] This is illustrated schematically with multiplex circuit MUX 1411, i.e., a controller, that receives the bit line addresses and also the ZCI values from the ZCI register 1420. If the selected bit line address matches a bit line with ZCI=0, the MUX 1411 can notify the bit line activation circuit, along with the ADC 1407 and shift and add 1409 so that the corresponding column can just be skipped in the sensing operation.};
Hoang ‘235 doesn’t explicitly disclose, however, Santoro, in a similar field of endeavor directed to augmenting neural networks with external memory, teaches: [perform task-specific inference] without reprogramming the memory cells {[0063] Alternatively, the system may maintain the external memory without deleting values even between tasks.}.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Hoang ‘235 to include the features of Santoro, in order to leverage data stored in the memory while processing a previous input sequence or even while performing a previous machine learning task {[0063] of Santoro}.
The combination of Hoang ‘235 and Santoro doesn’t explicitly teach, however, Mallya, in a similar field of endeavor directed to adapting a single network to multiple tasks by learning to mask weights, teaches: wherein the binary mask for each task is learned to selectively enable or disable columns for that task while keeping backbone model fixed {“Approach,” page 4: The key idea behind our method is to learn to selectively mask the fixed weights of a base network, so as to improve performance on a new task. We achieve this by maintaining a set of real-valued weights that are passed through a deterministic thresholding function to obtain binary masks, that are then applied to existing weights. By updating the real-valued weights through backpropagation, we hope to learn binary masks appropriate for the task at hand. This process is illustrated in Figure 1. By learning different binary-valued {0,1} masks per task, which are element-wise applied to network parameters, we can re-use the same underlying base network for multiple tasks, with minimal overhead. Even though we do not modify the weights of the network, a large number of different filters can be obtained through masking. For example, a dense weight vector such as [0.1,0.9,−0.5,1] can give rise to filters such as [0.1,0,0,1],[0,0.9,−0.5,0], and [0,0.9,−0.5,1] after binary masking. In practice, we begin with a network such as the VGG-16 or ResNet-50 pre-trained on the ImageNet classification task as our base network, referred to as the backbone network, and associate a real-valued mask variable with each weight parameter of all the convolutional and fully-connected layers. By combining techniques used in network binarization [26,27] and pruning [30], we train these mask variables to learn the task at hand in an end-to-end fashion, as described in detail below. The choice of the initialization of the backbone network is crucial for obtaining good performance, and is further analyzed in Section 5.1.}
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the combination of Hoang ‘235 and Santoro to include the features of Mallya, in order to improve performance on a new task {“Approach,” page 4 of Mallya}.
Claim 3
Hoang ‘235 further discloses: wherein the random-access memory comprises resistive random-access memory {[0040] Other examples of suitable technologies for memory cells of the memory structure 326 include ReRAM memories (resistive random access memories).
Claim 6
Hoang ‘235 further discloses: wherein each memory cell stores 2 bits {[0098] The use of the ZCI and ZRI bits allows for the elimination of unnecessary accesses to bit lines or word lines that contain all-zero weight values by deactivating their associated input/out, improving both performance and energy efficiency of CNN inference with sparse matrix multiplication.}.
Claim 8
Hoang ‘235 further discloses: wherein the binary mask buffer memory has a size of at least 2N bits, and is configured to store two separate masks of size N, each bit of each mask corresponding to a column in the subarray {[0065] For this purpose, a Zero Column Index (ZCI) 1420 is added at the array or sub-array level, adding one vector per array or sub-array. The size of the ZCI 1420 is the number of bit lines in the array or subarray 1401, which can be the same as the corresponding row buffer. Each bit line BL.sup.i has an entry in ZCI 1420 where, in this embodiment, ZCI.sup.i=‘0’ indicates the i.sup.th column having all-zero weights and ZCI.sup.i=‘1’ indicates the i.sup.th column having at least one non-zero weight, i.e., the binary mask buffer memory has a size of at least 2N bits.}.
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hoang ‘235, Santoro, and Mallya, further in view of Hou (US 20220164630).
Claim 4
The combination of Hoang ‘235, Santoro, and Mallya doesn’t explicitly teach, however, Hou, in a similar field of endeavor directed to deep convolutional neural networks, teaches: a real-valued mask buffer configured to store a calculated real-valued mask {[0044] The present disclosure provides a MIMO strategy in the 3D separable CNN. While existing networks are SISO, MISO, or two-input two-output, the MIMO network provided in the present disclosure can take multiple input frames and output multiple binary masks using temporal-dimension in each sample.}; and a sigmoid element configured to convert the real-valued mask into a binary mask for storage in the binary mask buffer memory {[0086] Finally in block 8, the feature maps are projected to a 4D output of size 6×H×W×1, and a sigmoid activation function is appended to generate the probability masks for 6 successive frames.}.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the combination of Hoang ‘235, Santoro, and Mallya to include the features of Hou, in order to facilitate faster inference speed and smaller model size while maintaining high detection accuracy {[0006] of Hou}.
Claim 5
Hou further teaches: wherein the real-valued mask buffer comprises floating-point values and the sigmoid element is a thresholding element having a threshold of 0.5 {[0086] A threshold of 0.5 is applied to convert the probability masks to binary masks that indicate the detected moving objects.}. The motivation and rationale to include the additional features of Hou is the same as set forth previously.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hoang ‘235, Santoro, and Mallya, further in view of Hoang (US 20210397974; hereinafter Hoang ‘974).
Claim 7
The combination of Hoang ‘235, Santoro, and Mallya doesn’t explicitly teach, however, Hoang ‘974, in a similar field of endeavor directed to a deep neural network engine, teaches: a plurality of N/2 shift-adders, each configured to combine two 2-bit weights from adjacent columns of the subarray into a 4-bit partial sum activation {[0106] FIG. 21 is a block diagram for one embodiment of the PSMA block 1363 of FIG. 13. As in the embodiment for cascaded adders of PPMA in FIG. 18, PSMA 1701 again includes a set of four (N/4)-bit adders, Adder-3 2101-3, Adder-2 2101-2, Adder-1 2101-1, and Adder-0 2101-0 that can be operated as four (N/4)-bit adders or connected into two (N/2)-bit adders or one N-bit adders based upon which of the connections of blocks 2105-2, 2105-1, and 2105-0 are enabled or disabled.}.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the combination of Hoang ‘235, Santoro, and Mallya to include the features of Hoang ‘974, in order to allow for a compute-in-memory inference engine that balances programmability, performance, and compute efficiency {[0079] of Hoang ‘974}.
Claims 10 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Hoang ‘235 in view of Santoro.
Claim 10
Hoang ‘235 discloses:
A method of machine learning for multiple task adaptation {[0022] When a convolution neural network (CNN) performs an inference operation, i.e., machine learning for multiple task adaptation, the most time consuming parts of the inference are the convolution operations as these are very computationally intensive matrix multiplication operations using large amounts of data. The convolutions, or matrix multiplications, are performed using sets of weights, referred to as filters, determined during a training process for the CNN. method described in [0101].}, comprising:
loading a backbone model into a random-access memory, the random-access memory comprising a plurality of subarrays, each subarray comprising M rows and N columns of memory cells, wherein each column of the N columns is configured to store a convolution kernel of the backbone model {[0063] FIG. 14 is an embodiment for an architecture that can leverage all-zero columns of a storage class memory sub-array to reduce the number of bit line accesses to improve performance and energy efficiency. FIG. 14 illustrates an array, or portion of an array, 1401 of resistive non-volatile memory cells and peripheral elements, similar to the portions of the array shown in FIG. 9, but with the memory cells represented as blocks at the intersection of word lines and bit lines. N bit lines, running from BL.sup.0 to BL.sup.N−1, and M word lines, running from WL.sup.0 to WL.sup.M−1, are shown and can represent the whole of an array or a compact portion of a larger array, i.e., loading a backbone model into a memory, the memory comprising a plurality of subarrays, each subarray comprising M rows and N columns of memory cells, wherein each column of the N columns is configured to store a convolution kernel of the backbone model. random-access memory described in [0050]: For example, on a host processor executing the neural network, the weight could be read out of an SSD in which they are stored and loaded into RAM on the host device.};
selecting a set of tasks to run on the backbone model, each task having a corresponding binary mask configured to enable or disable each of the N columns of the subarray {[0065] For this purpose, a Zero Column Index (ZCI) 1420 is added at the array or sub-array level, adding one vector per array or sub-array. The size of the ZCI 1420 is the number of bit lines in the array or subarray 1401, which can be the same as the corresponding row buffer. Each bit line BL.sup.i has an entry in ZCI 1420 where, in this embodiment, ZCI.sup.i=‘0’ indicates the i.sup.th column having all-zero weights and ZCI.sup.i=‘1’ indicates the i.sup.th column having at least one non-zero weight, i.e., selecting a set of tasks to run on the backbone model, each task having a corresponding binary mask configured to enable or disable each of the N columns of the subarray.};
selecting one task of the set of tasks and applying the binary mask corresponding to the task to the N columns of the subarray, disabling at least one column of the subarray {[0067] This is illustrated schematically with multiplex circuit MUX 1411 that receives the bit line addresses and also the ZCI values from the ZCI register 1420. If the selected bit line address matches a bit line with ZCI=0, the MUX 1411 can notify the bit line activation circuit, along with the ADC 1407 and shift and add 1409 so that the corresponding column can just be skipped in the sensing operation, i.e., selecting one task of the set of tasks and applying the binary mask corresponding to the task to the N columns of the subarray, disabling at least one column of the subarray.}; and
executing the task on the backbone model, ignoring the disabled convolution kernel to calculate a result {See [0067].}.
Hoang ‘235 doesn’t explicitly disclose, however, Santoro, in a similar field of endeavor directed to augmenting neural networks with external memory, teaches: [calculate a result] without reprogramming the memory cells {[0063] Alternatively, the system may maintain the external memory without deleting values even between tasks.}.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Hoang ‘235 to include the features of Santoro, in order to leverage data stored in the memory while processing a previous input sequence or even while performing a previous machine learning task {[0063] of Santoro}.
Claim 13
Hoang ‘235 further discloses: wherein the random-access memory is a resistive random-access memory {[0040] Other examples of suitable technologies for memory cells of the memory structure 326 include ReRAM memories (resistive random access memories).}.
Claims 11 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hoang ‘235 and Santoro, further in view of Hou.
Claim 11
The combination of Hoang ‘235 and Santoro doesn’t explicitly teach, however, Hou, in a similar field of endeavor directed to deep convolutional neural networks, teaches: calculating real-valued masks to correspond to each task in the set of tasks {[0044] The present disclosure provides a MIMO strategy in the 3D separable CNN. While existing networks are SISO, MISO, or two-input two-output, the MIMO network provided in the present disclosure can take multiple input frames and output multiple binary masks using temporal-dimension in each sample.}; and calculating the corresponding binary masks from the real-valued masks with a sigmoid function {[0086] Finally in block 8, the feature maps are projected to a 4D output of size 6×H×W×1, and a sigmoid activation function is appended to generate the probability masks for 6 successive frames.}. The motivation and rationale to include the additional features of Hou is the same as set forth previously.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the combination of Hoang ‘235 and Santoro to include the features of Hou, in order to facilitate faster inference speed and smaller model size while maintaining high detection accuracy {[0006] of Hou}.
Claim 14
The combination of Hoang ‘235 and Santoro doesn’t explicitly teach, however, Hou, in a similar field of endeavor directed to deep convolutional neural networks, teaches: calculating a first partial sum in a first subarray of the plurality of subarrays, and a second partial sum in a second subarray of the plurality of subarrays {[0047] For example, the model in the running average background dynamically updates the background image to adapt to the scene changes by computing the weighted sum of the current frame and the previously estimated background image.}; and combining the first and second partial sums to calculate an activation {See [0047].}.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the combination of Hoang ‘235 and Santoro to include the features of Hou, in order to facilitate faster inference speed and smaller model size while maintaining high detection accuracy {[0006] of Hou}.
Response to Arguments
Applicant's arguments filed 5/26/26 have been fully considered.
Applicant’s is thanked for their amendments overcoming the previous rejections under 35 U.S.C. §112. These are withdrawn, in view of the renaming of “volatile memory” to “random-access memory.”
With respect to applicant’s arguments concerning the rejections under 35 U.S.C. §103, examiner notes that they are predicated on the present claim amendments (claim 7, which depends on claim 1, being modified by virtue of its dependency). These arguments are moot, in view of the new grounds of rejection that’s necessitated by amendment. Examiner directs applicant to the claim analysis above.
In summary, examiner has responded to all remarks.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 20160232951, which teaches: A compute memory system can include a memory array and a controller that generates N-ary weighted (e.g., binary weighted) access pulses for a set of word lines during a single read operation. This multi-row read generates a charge on a bit line representing a word stored in a column of the memory array. The compute memory system further includes an embedded analog signal processor stage through which voltages from bit lines can be processed in the analog domain. Data is written into the memory array in a manner that stores words in columns instead of the traditional row configuration.
US 20180046915, which teaches: The present invention relates to artificial neural networks, for example, deep neural networks. In particular, the present invention relates to a compression method for deep neural networks with proper use of mask and the device thereof. More specifically, the present invention relates to how to compress dense neural networks into sparse neural networks while maintaining or even improving the accuracy of the neural networks after compression.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN SAMUEL WASAFF whose telephone number is (571)270-5091. The examiner can normally be reached Monday through Friday 8:00 am to 6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SARAH MONFELDT can be reached at (571) 270-1833. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JOHN SAMUEL WASAFF
Primary Examiner
Art Unit 3629
/JOHN S. WASAFF/Primary Examiner, Art Unit 3629
Prosecution Insights