DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1- 30 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more.
Regarding Claim 1:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
generate a converted set of parameters based on application of a conversion operation to format the set of parameters according to a second encoding format; This limitation is directed to the abstract idea of a mathematical concepts, as generating a set of parameters is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
generate a set of bit planes based on application of a bit plane transformation to the converted set of parameters; This limitation is directed to the abstract idea of a mathematical concepts, as generating a set of bit plane based on bit plane transformation is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
generate a compressed set of parameters for the machine learning model based on application of a bit mask operation to one or more bit planes of the set of bit planes. This limitation is directed to the abstract idea of a mathematical concepts, as generating a set of parameters based on mask operation is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
A processing system for data compression, comprising: a memory comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to: This limitation recites generic computer components such as processor and memory, which invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to integrate the exception into a practical application.
access a set of parameters for a machine learning model, wherein the set of parameters are formatted according to a first encoding format; This limitation recites as an insignificant extra solution activity, as accessing parameters from devices, under BRI, is mere data gathering per MPEP 2106.05(g)(3).
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
A processing system for data compression, comprising: a memory comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to. This limitation invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to amount to significantly more than the judicial exception.
access a set of parameters for a machine learning model, wherein the set of parameters are formatted according to a first encoding format; This limitation is directed to receiving data, which the courts have recognized as well-understood, routine, conventional activity when they are claimed at a high level of generality or as insignificant extra-solution activity [see MPEP 2106.05(d) II. i] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 2:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
for each respective bit plane of the set of bit planes, determine a respective word length to encode the respective bit plane. This limitation is directed to the abstract idea of a mental process as determining a word length is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding Claim 3:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
encode a first bit plane of the set of bit planes based on a first word length; This limitation is directed to the abstract idea of a mathematical concepts, as encoding a bit plane is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
determine a first sparsity value of the first bit plane encoded based on the first word length; This limitation is directed to the abstract idea of a mathematical concepts, as determining a sparsity value is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.) and determining a sparsity value which is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
encode the first bit plane based on a second word length; This limitation is directed to the abstract idea of a mathematical concepts, as encoding a bit plane is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
determine a second sparsity value of the first bit plane encoded based on the second word length; This limitation is directed to the abstract idea of a mathematical concepts, as encoding a bit plane is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.) and determining a sparsity value which is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
select the first word length for the first bit plane based on the first and second sparsity values. This limitation is directed to the abstract idea of a mental process selecting the word length for bit plane is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding Claim 4:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
determine to apply the bit mask operation to a first bit plane of the set of bit planes, based on a first sparsity value of the first bit plane; This limitation is directed to the abstract idea of a mental process as determining to apply the bit mask operation is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
determine to not apply the bit mask operation to a second bit plane of the set of bit planes, based on a second sparsity value of the second bit plane. This limitation is directed to the abstract idea of a mental process as determining to apply the bit mask operation is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding Claim 5:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
generate a pruned set of bit planes based on pruning of one or more words used to encode the set of bit planes based on a magnitude threshold, wherein the magnitude threshold is a hyperparameter; This limitation is directed to the abstract idea of a mathematical concepts, as generating a pruned set of bit planes is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
decode the pruned set of bit planes to generate a pruned set of parameters; This limitation is directed to the abstract idea of a mathematical concepts, as decoding the pruned set of bit planes is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
update one or more parameters of the pruned set of parameters using training data; This limitation is directed to the abstract idea of a mental process as updating parameters is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
encode the pruned set of parameters based on application of the conversion operation, the bit plane transformation, and the bit mask operation to generate a pruned compressed set of parameters. This limitation is directed to the abstract idea of a mathematical concepts, as encoding the pruned set of bit planes is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
Regarding Claim 6:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
determine that the one or more words have a magnitude smaller than the magnitude threshold; This limitation is directed to the abstract idea of a mathematical concepts, as comparing the magnitude with the threshold is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.) and comparing a magnitude with the threshold which is analogous to a mental process. (see MPEP 2106.04 (a)(2) III.A).
set each of the one or more words to a value of zero. This limitation is directed to the abstract idea of a mental process as set words to a value of zero is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding Claim 7:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
determine a mode of a set of words used to encode the first bit plane; This limitation is directed to the abstract idea of a mental process as determining a mode is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
compress the first bit plane based on the mode. This limitation is directed to the abstract idea of a mathematical concepts, as compressing the bit plane is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
identify one or more words of the set of words that have values matching the mode; This limitation is directed to the abstract idea of a mental process as identifying words that have values matching the mode is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
replace each respective word of the one or more words with a respective mask bit indicating that the respective word has a value equal to the mode. This limitation is directed to the abstract idea of a mental process as replacing the words with a mask bit is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding Claim 8:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
identify one or more parameters of the set of parameters that have a value of zero; This limitation is directed to the abstract idea of a mental process as identifying parameters have a value of zero is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
encode each respective parameter of the one or more parameters without including a respective sign bit, wherein one or more other parameters having non-zero values are encoded with sign bits. This limitation is directed to the abstract idea of a mathematical concepts, as encoding parameters is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
Regarding claims 9- 12
Claims 9 - 12 recites analogous limitations to claims 1 -4 (respectively) and therefore they are rejected on the same grounds as claims 1 - 4.
Regarding claims 13 - 14
Claims 13 - 14 recites analogous limitations to claims 7 - 8 (respectively) and therefore they are rejected on the same grounds as claims 7 - 8.
Regarding claims 15 - 22
Claims 15 - 22 recites analogous limitations to claims 1 - 8 (respectively) and therefore they are rejected on the same grounds as claims 1 - 8.
Regarding claims 23 - 30
Claims 23 - 30 recites analogous limitations to claims 1 - 8 (respectively) and therefore they are rejected on the same grounds as claims 1 - 8.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 – 4, 7, 9 – 13, 15 – 18, 21, 23 – 26 and 29 are rejected under 35 U.S.C. 102(a) (1) as being anticipated by Cavigelli (NPL, EBPC: Extended Bit-Plane Compression for Deep Neural Network Inference and Training Accelerators) dated on 10/25/2019, by Cavigelli et al - hereinafter Cavigelli).
Referring to Claim 1, Cavigelli teaches:
A processing system for data compression, comprising: a memory comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to: See Cavigelli at [Page 1, Abstract]:” In the wake of the success of convolutional neural networks in image classification, object recognition, speech recognition, etc., the demand for deploying these compute-intensive ML models on embedded and mobile systems with tight power and energy constraints at low cost, as well as for boosting throughput in data centers, is growing rapidly. This has sparked a surge of research into specialized hardware accelerators. Their performance is typically limited by I/O bandwidth, power consumption is dominated by I/O transfers to off-chip memory, and on-chip memories occupy a large part of the silicon area.” Examiner interprets the embedded and mobile systems, the on-chip/off-chip memory as equivalent as the system and the computer components as claimed.
access a set of parameters for a machine learning model, wherein the set of parameters are formatted according to a first encoding format; See Cavigelli at [Page 1, mid - right ]:” In contrast to these, the focus on this paper is on reducing the energy consumption of hardware accelerators for CNN inference and training by cutting down on the dominant power contributor—I/O transfers. These data transfers to and from off-chip memory consist of the network parameters (read-only) and the feature maps (read/write).” Examiner interprets transferring data consist of the network parameters to/from the memory as equivalent as accessing parameters for a model. Also, see Cavigelli at [Page 4, III.b]:” An overview of the bit-plane compressor (BPC) used to compress the non-zero values is shown in Fig. 1. For BPC a set of n words of m bit, a data block, is compressed by first computing differences between every two consecutive words and storing the first word as the base.” Examiner interprets a set of n words of m bit, a data block as equivalent as a first encoding format. Thus, Cavigelli teaches the limitation.
generate a converted set of parameters based on application of a conversion operation to format the set of parameters according to a second encoding format; See Cavigelli at [Page 4, top - left]:” For BPC a set of n words of m bit, a data block, is compressed by first computing differences between every two consecutive words and storing the first word as the base. This exploits that neighboring values are often similar to reduce concentrates the distribution of the compressed values around zero.” Examiner interprets compressing data by first computing differences as equivalent as generating a converted set of parameters; the format of the combination of base (first word) and following difference (difference between two consecutive words) as equivalent as the second encoding format.
generate a set of bit planes based on application of a bit plane transformation to the converted set of parameters; See Cavigelli at [Page 4, top - left]:” The data items storing these differences are then viewed as m + 1 bit-planes of n -1 1 bit each (delta bit-planes, DBPs). Neighboring DBPs are XOR-ed, now called DBX, and the DBP of the most significant bit is kept as the base-DBP.” Examiner interprets storing and viewing the data items as m+1 bit-planes of n -1 bit each (delta bit – plane, DBPs) as equivalent as generating a set of bit planes.
generate a compressed set of parameters for the machine learning model based on application of a bit mask operation to one or more bit planes of the set of bit planes. See Cavigelli at [Page 4, mid - left]:” The results are fed into bit-plane encoders, which compress the DBX and DBP values to a bit-stream following Table 3a.” Examiner interprets the bit-plane encoders compressing the DBX and DBP values as equivalent as generating a compressed set of parameters. Also, see Cavigelli at [Page 2, II.A.(4)]:” 4) Zero-value compression (ZVC) (first used in this context in NullHop [11], then in cDMA [25]): “Zero-value compression (ZVC) (first used in this context in NullHop [11], then in cDMA [25]): Saves a fixed-length mask indicating whether a value was zero or nonzero and a variable-length list of the non-zero values.” Examiner interprets using fixed-length mask method as equivalent as application of a bit mask operation. Thus, Cavigelli teaches the limitation.
Referring to Claim 2, Cavigelli teaches:
for each respective bit plane of the set of bit planes, determine a respective word length to encode the respective bit plane. See Cavigelli at [Page 5, IV.A]:” The data block is then read by the DBP/DBX encoder to encode each bit-plane as a bit-vector and its length. The resulting variable-length data is then packed with a circuit similar to the packer in the Zero-RLE block to produce fixed 8 bit length words.” Examiner interprets encoding each bit-plane as a bit-vector and packing with the Zero-RLE block to produce fixed 8-bit length words as equivalent as determining a word length to encode the respective bit plane.
Referring to Claim 3, Cavigelli teaches:
encode a first bit plane of the set of bit planes based on a first word length; See Cavigelli at [Page 5, IV.A]:” The data block is then read by the DBP/DBX encoder to encode each bit-plane as a bit-vector and its length. The resulting variable-length data is then packed with a circuit similar to the packer in the Zero-RLE block to produce fixed 8 bit length words.” Examiner interprets the encoder encoding each bit-plane as a bit-vector and its length as equivalent as encoding a first bit plane based of a first word length.
determine a first sparsity value of the first bit plane encoded based on the first word length; See Cavigelli at [Page 3, mid - right]:” Sparsity: The value stream is decomposed into a zero/non-zero stream on which we apply run-length encoding to compress the zero burst commonly occurring in the data.” Also, see Cavigelli at [Page 3, bottom - right]:” Bursts of zeros are encoded by a ’0’ bit followed by a fixed number of bits describing the length of the burst.” Cavigelli disclosed exploiting sparsity by decomposing the value stream into a zero/non-zero stream and applying run-length encoding to compress the zero burst commonly occurring in the data. Since bursts of zeros are encoded by bits describing the length of the burst, Examiner interprets the zero-burst length constitutes a sparsity value under the broadest reasonable interpretation (BRI). Examiner interprets the first sparsity value is determined when a plane was encoded with a given word length/width and a given zero-burst length.
encode the first bit plane based on a second word length; determine a second sparsity value of the first bit plane encoded based on the second word length; See Cavigelli at [Page 9, top – right, Table III]:
PNG
media_image1.png
228
608
media_image1.png
Greyscale
In table III, Cavigelli discloses three different word width (8, 16, 32) as rows and six different zero burst length as columns (21, 22,…, 26), and different compression ratios are based on these values. Examiner interprets using different word width and zero burst length to compress the data as equivalent as encoding a plane based on a second word length, and also the second sparsity value is determined as different zero burst length is given. Thus, Cavigelli teaches the limitations.
select the first word length for the first bit plane based on the first and second sparsity values. See Cavigelli at [Page 9, bottom - left]:” Max. Zero Burst Length: We first analyze the effect of varying the maximum zero burst length for Zero-RLE on the compression ratio without for various data word widths in Table III. The optimal value is arguably the same for our proposed method, since a constant offset in compressing the non-zero values does not affect the optimal choice of this parameter (just like the word width has no effect on it).” As the examiner interpreted above, Cavigelli evaluates different zero burst lengths, which constitutes first and second sparisity values under the broadest reasonable interpretation, and determines the resulting compression ratios associated with those sparsity values. Cavigelli further identifies an optimal value based on the evaluated compression ratios. Because the compression ratios are determined from the evaluated sparsity values, the identified optimal value is selected based on the first and second sparsity values. Thus, selecting the word length based on the optimal value as equivalent as selecting the (first or second) word length based on (first or second) sparsity values.
Referring to Claim 4, Cavigelli teaches:
determine to apply the bit mask operation to a first bit plane of the set of bit planes, based on a first sparsity value of the first bit plane; and determine to not apply the bit mask operation to a second bit plane of the set of bit planes, based on a second sparsity value of the second bit plane. See Cavigelli at [Page 2, mid - right]:” There are several publications in literature describing hardware accelerators which exploit feature map sparsity to reduce computation: Cnvlutin [8], SCNN [9], Cambricon-X [10], NullHop [11], Eyeriss [12], EIE [13]. Their focus is on power gating or skipping some of the operations and memory accesses. This entails defining a scheme to feed the data into the system. They all use one of four methods … 4) Zero-value compression (ZVC) (first used in this context in NullHop [11], then in cDMA [25]): Saves a fixedlength mask indicating whether a value was zero or nonzero and a variable-length list of the non-zero values.” Examiner interprets method (4) Zero-value compression (ZVC): saving a fixed-length mask as equivalent as the bit mask operation since it is well known in the art that sparsity is determined by the quantity and distribution of zeros within a matrix. Consequently, the decision to apply or not apply a bit mask operation or an alternative technique is inherently based on these sparsity values. Thus, Cavigelli teaches this exact logic: for bit planes with high sparsity, the system applies a “fixed-length mask”, and for bit planes with low sparsity, the system determines not to use a mask.
Referring to Claim 7, Cavigelli teaches:
determine a mode of a set of words used to encode the first bit plane. See Cavigelli at [Page 5, Fig. 3]:
PNG
media_image2.png
390
1262
media_image2.png
Greyscale
As Fig. 3 shows, Cavigelli discloses analyzing DBX/DBP symbol patterns used to encode bit planes and provides a symbol histogram showing the frequency of those patterns. The most frequently occurring DBX/DBP symbol pattern in the set of encoded bit-plane words as equivalent as a mode since the mode number is defined as the most frequent number/word(s) in the dataset in the field of mathematics and statistics. Thus, Cavigelli teaches the limitation.
compress the first bit plane based on the mode:
identify one or more words of the set of words that have values matching the mode; See Cavigelli at [page 5, Fig. 3]: As Fig 3 shows, assuming the all-zero DBX/DBP pattern is the determined mode, Cavigelli discloses identifying words matching that mode because its bit-plane encoder detects all-zero DBX/DBP symbols and assigns them corresponding short code symbols, such as the “multi-all-0 DBX” and “all -0 DBX” entries in Fig. 3(a) Therefore, Cavigelli teaches identifying words of the set of encoded bit-plane words that match the determined mode.
replace each respective word of the one or more words with a respective mask bit indicating that the respective word has a value equal to the mode. See Cavigelli at [Page 2, bottom - right]:”(4) Zero-value compression (ZVC) (first used in this context in NullHop [11], then in cDMA [25]): Saves a fixed length mask indicating whether a value was zero or nonzero and a variable-length list of the non-zero values.” Also, see Cavigelli at [Page 3, bottom - left]:” They useZVC in a configuration which takes a block of 32 activation values and generates a 32-bit mask where only the bits to the non-zero values are set. The non-zero values are stored and transferred after the masks.” Cavigelli discloses ZVC, which saves a fixed–length mask indicating thether each value was zero or non-zero, and further discloses a 32- value block producing a 32-bit mask, with non-zero values stored after the masks. Because each value in the block corresponds to a respective bit of the 32-bit mask, and because an unset/set mask bit indicates whether that value is zero or non-zero, Cavigelli teaches replacing each respective value/word with a respective mask bit. Under BRI, where the determined mode is the zero value or all-zero word, the mask bit indicates whether the respective word has a value equal to the mode.
With the combination of the identifying the words that matching the mode and replacing the words with the mask bit, Cavigelli teaches “compressing the plane based on the mode” as claimed.
Referring to claim(s) 9 - 12, 15 – 18, 23- 26, the claim(s) is/are rejected on the same basis as claim(s) 1 - 4, mutatis mutandis, since they are analogous claims.
Referring to claim(s) 13, 21, 29, the claim(s) is/are rejected on the same basis as claim(s) 7, mutatis mutandis, since they are analogous claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claim(s) 5 – 6, 19 – 20 and 27 - 28 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cavigelli in view of Han (NPL, Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, dated on 02/15/2016, by Han et al - hereinafter Han).
Referring to Claim 5, Cavigelli teaches the system of claim 1, However, it fails to teach:
generate a pruned set of bit planes based on pruning of one or more words used to encode the set of bit planes based on a magnitude threshold, wherein the magnitude threshold is a hyperparameter; decode the pruned set of bit planes to generate a pruned set of parameters; update one or more parameters of the pruned set of parameters using training data; encode the pruned set of parameters based on application of the conversion operation, the bit plane transformation, and the bit mask operation to generate a pruned compressed set of parameters.
Han teaches, in an analogous system:
generate a pruned set of bit planes based on pruning of one or more words used to encode the set of bit planes based on a magnitude threshold, wherein the magnitude threshold is a hyperparameter; See Han at [Page 2]:” As shown on the left side of Figure 1, we start by learning the connectivity via normal network training. Next, we prune the small-weight connections: all connections with weights below a threshold are removed from the network.” Han discloses pruning based on magnitude because connections having weights below a threshold are removed from the network. And, See Han at [Page 3, Figure 3], Han further discloses representing low-magnitude weight values by a zero-valued centroid during compression. Under the broadest reasonable interpretation (BRI), replacing low-magnitude words with a zero value as equivalent as pruning of the words. Also, Examiner interprets the threshold in Han is as equivalent to the hyperparameter as claimed since a POSITA would have understood the threshold to be a user-selected turning parameter that controls which weights are pruned and is not itself a learned network parameter. Therefore, Han teaches generating a pruned set of encoded data values based on a magnitude threshold.
PNG
media_image3.png
502
946
media_image3.png
Greyscale
decode the pruned set of bit planes to generate a pruned set of parameters; See Cavigelli at [Page 5, bottom - left]:” The decompressor shown in Fig. 5 reverts the steps of the encoder. After inverting the Zero-RLE encoding, the bitplane compressed data stream is read in 8 bit words, and unpacked into variable-length data chunks. The Unpacker always provides 8 valid data bits to the Symbol Decoder, which decodes the symbol into a DBP or DBX word and feeds the effective symbol length back to the Unpacker.” Examiner interprets the decoder decodes the data of the bit plane and feeds the symbol length back to the unpacker as equivalent as decoding the bit plane to generating set of parameters, as variable-length data is considered as the parameter data under BRI. Thus, applying the combination with the pruned parameters that Han teaches in the limitation above, Cavigelli-Han teaches the limitation.
update one or more parameters of the pruned set of parameters using training data; See Han at [Page 5, top]:” During back-propagation, the gradient for each shared weight is calculated and used to update the shared weight. This procedure is shown in Figure 3.” Examiner interprets updating the shared weight as equivalent as updating the parameters since a POSITA would understand that the gradient-based update occurs using training data during back-propagation (a method to train a model). Thus, Han teaches the limitation.
encode the pruned set of parameters based on application of the conversion operation, the bit plane transformation, and the bit mask operation to generate a pruned compressed set of parameters. See Han at [Page 2, bottom]:” Next, we prune the small-weight connections: all connections with weights below a threshold are removed from the network. Finally, we retrain the network to learn the final weights for the remaining sparse connections.” Han discloses pruning small-weight conections and retraining the remaining sparse connections, which is as equivalent as encoding and generating a pruned set of parameters. Also, as Examiner interprets in claim 1 – claim 4, Cavigelli teaches encoding data words by applying the conversion operation, bit-plane transformation, and bit-mask operation to generate a compressed bit-stream. Therefore, applying Cavigelli’s previously discussed encoding process to Han’s pruned parameters, Cavigelli - Han teaches the limitation.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Cavigelli with the above teachings of Han by decoding the pruned set of bit planes to generate a pruned set of parameters, as taught by Cavigelli, pruning the words, encoding the parameters, and generating a pruned compressed set of parameters, as taught by Han. The modification would have been obvious because one of ordinary skill in the art would be motivated to reduce the storage requirement of neural networks without affecting the accuracy, See Han at [Page 1, Abstract]:” To address this limitation, we introduce “deep compression”, a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy.”
Referring to Claim 6, Cavigelli teaches the system of claim 1. However, it fails to teach:
determine that the one or more words have a magnitude smaller than the magnitude threshold; and set each of the one or more words to a value of zero.
Han teaches, in an analogous system:
determine that the one or more words have a magnitude smaller than the magnitude threshold; See Han at [Page 2, bottom]:” all connections with weights below a threshold are removed from the network.” Examiner interprets removing the connections with weights below a threshold as equivalent as determining one or more words having magnitude smaller than the magnitude threshold.
set each of the one or more words to a value of zero. See Han at [Page 3, Figure 3, two matrix on the first line]: As Figure 3 shows, in left matrix, the value “-0.98”, “-1.08”, “-0.91” and “-1.03” that below zero are replaced as a value zero on the right matrix, which is as equivalent as setting the words to a value of zero under BRI.
PNG
media_image4.png
236
428
media_image4.png
Greyscale
The same motivation that was utilized for combining Cavigelli with Han as set forth in claim 5 is equally applicable to claim 6.
Referring to claim(s) 19 – 20 and 27 - 28, the claim(s) is/are rejected on the same basis as claim(s) 5 - 6, mutatis mutandis, since they are analogous claims.
Claim(s) 8, 14, 22 and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cavigelli in view of Wiedemann (NPL, DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks, dated on 07/27/2019, by Wiedemann et al - hereinafter Wiedemann).
Referring to Claim 8, Cavigelli teaches the system of claim 1. However, it fails to teach:
identify one or more parameters of the set of parameters that have a value of zero; encode each respective parameter of the one or more parameters without including a respective sign bit, wherein one or more other parameters having non-zero values are encoded with sign bits.
Wiedemann teaches, in an analogous system:
identify one or more parameters of the set of parameters that have a value of zero; See Wiedemann at [Page 7, bottom - right]:” The first bit, sigFlag, determines if the weight element is a significant element or not. That is, it indicates if the weight value is 0 or not.” Examiner interprets indicating if the weight value is 0 or not as equivalent as identifying parameters that have a value of zero.
encode each respective parameter of the one or more parameters without including a respective sign bit, wherein one or more other parameters having non-zero values are encoded with sign bits. See Wiedemann at [Page 7, bottom-right]:” Then, if the element is not 0, the sign bit or signFlag is analogously encoded, according to its respective context model.” Examiner interprets encoding sign bit for non-zero element as equivalent as encoding the parameters having non-zero values with sign bit.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Cavigelli with the above teachings of Wiedemann by the compressing system, as taught by Cavigelli, identifying whether the parameters have a value of zero, and encoding the parameters with the sign bits, as taught by Wiedemann. The modification would have been obvious because one of ordinary skill in the art would be motivated to consistently attain higher compression rates, See Wiedemann at [Page 1, Abstract]:” Experimental results show that DeepCABAC consistently attains higher compression rates than previously proposed coding techniques for neural network compression. For instance, it is able to compress the VGG16 ImageNet model by x63.6 with no loss of accuracy, thus being able to represent the entire network with merely 8.7MB.”
Referring to claim(s) 14, 22 and 30, the claim(s) is/are rejected on the same basis as claim(s) 8, mutatis mutandis, since they are analogous claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIAYUE MA whose telephone number is (571)272-9658. The examiner can normally be reached between 9 am to 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jiayue Ma/
Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights