Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 7, 12, and 17 are objected to because of the following informalities: the claim recites kernel weights which haven’t been previously mentioned in the claim. There is insufficient antecedent basis for this term. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Regarding claim 1:
Step 2A Prong 1:
Connecting the selected one or more blocks with the selected operations to build the kernel reparameterization, wherein the kernel reparameterization has a dimension same as that of the convolution-wise operation kernel. (Mental process, connecting selected blocks given requirements can be done mentally or with pen and paper. The kernel dimensions must match so that data can be processed.)
Step 2A Prong 2:
A method of building a kernel reparameterization for replacing a convolution-wise operation kernel in training of a neural network, compromising:
Selecting one or more blocks from tensor blocks and operations; and (Adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – Examiner’s note: merely gathering tensor block data and operation data)
“neural network” (Adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses software as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level application of a neural network not specifying not specifying how actions are performed.)
“kernel ” (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level application of kernels or computer hardware to be reparametrized.)
Step 2B:
Selecting one or more blocks from tensor blocks and operations; and (Adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – Examiner’s note: merely gathering tensor block data and operation data) (Gathering block and operation data is well-understood and routine, see MPEP 2106.05(g))
“neural network” (Adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses software as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level application of a neural network not specifying not specifying how actions are performed.)
“kernel ” (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level application of kernels or computer hardware to be reparametrized.)
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception when taken alone or in combination. The additional elements are applications of the exception and mere insignificant extra solution activity being implemented with generic computer elements and software.
The claim is ineligible
Regarding claim 2:
The method of claim 1, wherein the tensor blocks compromise a 1x1 kernel, a 1xN kernel, an Nx1 kernel, an MxP kernel, an NxN kernel, and an identify kernel, wherein M ≤ N, P ≤ N, and each M, N, and P is a natural number.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 2, under broadest reasonable interpretation, does not include any new abstract ideas.
The claim recites additional elements of adding insignificant extra-solution activity to the judicial
exception MPEP 2106.05(g) – (Examiner’s note: merely mentions what the kernels in the tensor data is)
The claim does not include additional elements that are sufficient to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just insignificant extra-solution activity. Specifying the tensor blocks doesn’t change the reparameterization; and when taken individually or in combination with previous additional elements, it is well understood and routine conventional activity that adds nothing meaningful to the exceptions. Therefore, the judicial exceptions are not integrated into a practical application.
The claim is ineligible.
Regarding claim 3:
The method of claim 1, wherein the selected operations compromise one or more operations add, convolution, concatenation, and element-wise
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 2, under broadest reasonable interpretation includes a new abstract idea. The claim recites a mental process, choosing specific operations necessary for processing data can be done mentally.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception when taken alone or in combination with previous claims. Therefore, the judicial exceptions are not integrated into a practical application.
The claim is ineligible.
Regarding claim 4:
The method of claim 3, wherein the kernel reparameterization is a linear combination of the selected one or more tensor blocks.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 3. The limitations of claim 4, under broadest reasonable interpretation, includes a new abstract idea. The claim recites a mental process, making a linear combination is just math and can be performed by mentally processing data.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception when taken alone or in combination with previous claims. Therefore, the judicial exceptions are not integrated into a practical application.
The claim is ineligible.
Regarding claim 5:
The method of claim 1, wherein the convolution-wise operation comprises one or more operations of convolution, deconvolution or transposed convolution, deformable convolution, depth-wise convolution, and grouped convolution, with any stride, dilation, and padding.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 5, under broadest reasonable interpretation includes a new abstract idea. The claim recites a mental process, identifying a specific operation for processing data can be done mentally. The convolutions listed are also abstract mathematical concepts.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception when taken alone or in combination with previous claims. Therefore, the judicial exceptions are not integrated into a practical application.
The claim is ineligible.
Regarding claim 6:
A kernel reparameterization built according to claim 1 used in training of the neural network.
This provides a further description of the method within the abstract ideas, as discussed with regards to claim 1. The limitations of claim 6, under broadest reasonable interpretation, does not include any new abstract ideas.
The claim recites additional elements of adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses software as a tool to perform an abstract idea, see MPEP 2106.05(f) – (Examiner’s note: High level application of training a neural network not specifying how actions are performed)
The claim does not include additional elements that are sufficient to significantly more than the judicial exceptions when taken alone or in combination with previous claims. The claimed additional element is just adding the words “apply it” (or an equivalent) and merely using software as a tool to perform an abstract idea, see MPEP 2106.05(f). This is a high level application of training a neural network not specifying not specifying how the action is performed. When taken individually or in combination with previous additional elements, the current additional element is well understood and routine conventional activity that adds nothing meaningful to the exceptions. Therefore, the judicial exceptions are not integrated into a practical application.
The claim is ineligible.
Regarding claim 7:
Step 2A Prong 1:
A method of performing quantization aware training (QAT) of a neural network, comprising:
(a) identifying a convolution-wise operation kernel in …; (Mental process, identifying a kernel which uses specifically convolution data can be in the mind by observation.)
(c) connecting the selected one or more blocks with the selected operations to build a kernel reparameterization, wherein the kernel reparameterization has a dimension same as that of the convolution-wise operation kernel; (Same rationale as claim 1. This is a mental process, connecting selected blocks specifically with the same dimensions can be done mentally.)
(d) replacing the convolution-wise operation kernel with the kernel reparameterization; (Mental process, replacing the kernels operation can be done mentally by evaluation.)
(e) adding fake quant operator right after the convolution-wise operation; and (Mathematical concept, adding quant operators to convolution kernels is just math.)
(f) performing quantization-aware-training of the neural network with the kernel reparameterization, wherein the kernel weight is calculated on-the-fly using step (c). (Mental process, calculating kernel weights and quantizing them can be done mentally though math.)
Step 2A Prong 2:
“training of the neural network” (Adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses software as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level application of training a neural network not specifying how the action is performed)
(b) selecting one or more blocks from tensor blocks and operations; (Same rationale as claim 1, adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – Examiner’s note: merely gathering tensor block data and operation data)
Step 2B:
“training of the neural network” (Adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses software as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level application of training a neural network not specifying how the action is performed)
(b) selecting one or more blocks from tensor blocks and operations; (Same rationale as claim 1, adding insignificant extra-solution activity to the judicial exception MPEP 2106.05(g) – Examiner’s note: merely gathering tensor block data and operation data) (Same rationale as claim 1, gathering block and operation data is well-understood and routine, see MPEP 2106.05(g))
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception when taken alone or in combination. The additional elements are mere applications of exceptions or are well-understood routine and conventional.
The claim is ineligible
Regarding claims 8 – 11:
Claims 8 – 11 are method claims reciting the same activities as listed by the limitations of claims 2 – 5. The claims are rejected for having the same judicial exceptions and additional elements. They do not include additional elements that amount to significantly more than the judicial exceptions.
The claims are ineligible.
Regarding claims 12 – 16:
Claims 12 - 16 are system claims reciting the same activities as listed by the limitations of claims 7 - 11. The claims are rejected for having the same judicial exceptions and additional elements. They do not include additional elements that amount to significantly more than the judicial exceptions.
The claims are ineligible.
Regarding claims 17 – 20:
Claims 17 – 20 are machine claims reciting the same activities as listed by the limitations of claims 7 - 10. The claims are rejected for having the same judicial exceptions and additional elements. They do not include additional elements that amount to significantly more than the judicial exceptions.
The claims are ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (US 20160335120 A1) in view of Ben-dror et al. (US 20220327386 A1).
Claim 1 recites:
A method of building a kernel reparameterization for replacing a convolution-wise operation kernel in training of a neural network, compromising:
Selecting one or more blocks from tensor blocks and operations; and
Gupta Fig 5A and [0065] teaches training a convolutional neural network containing layers of operations. Combined with Fig 5B, Gupta teaches each convolution layer operation is run by an individual kernel. [0076] teaches input and output tensors, also known as tensor blocks, and use of convolution layer operations as a filter for a kernel. in Fig 5A, these are the image, output, and convolutional layers 1-3 of the network. [0033 - 0034] teaches the kernel can be re-parameterized to select and perform different operations. This kernel can also be split into multiple other smaller kernels for processing if necessary.
Connecting the selected one or more blocks with the selected operations to build the kernel reparameterization, ….
Gupta [0033] teaches the reparameterization of a convolution kernel. [0034] teaches this kernel can be split into multiple other smaller kernels for processing if necessary. Fig 1B and 1C teach kernels connected together with each kernel having their won operations.
Gupta doesn’t teach “wherein the kernel reparameterization has a dimension same as that of the convolution-wise operation kernel.”
Ben-dror does teach “wherein the kernel reparameterization has a dimension same as that of the convolution-wise operation kernel.” ([0114] and Fig 12 - 15 teaches replacing multiple convolutional kernels with one convolutional kernel by convolution folding where the total field of view or dimensionality of the convolution is equal and maintained. This folding function is a method of kernel reparameterization by definition. Maintaining the same field of view means the input-tensor and output-tensor sizes, would be the same when combined with the previous multiple convolutions or the new single convolution. This relationship is applicable for the opposite as well where a single convolution is reparametrized to multiple convolutions.)
Gupta’s teaching and Ben-dror’s teaching are analogous art because both references concern updating convolution kernel layers with new convolution layers as per reparameterization. It would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Ben-dror’s teaching of maintaining the dimensions of a kernel or a group of kernels when reparametrized with Gupta’s teaching. Gupta [0033] teaches the user specifying the input feature size and the size of the convolution filter kernels. Gupta’s embodiment of specifying the dimensions of the reparametrized kernel or kernels can combined with Ben-dror’s teaching of maintaining the field of view across reparameterization. This is necessary for correct training of the neural network where the input and output sizes of each layer are specific and necessary to continue.
Regarding claim 3:
The method of claim 1, wherein the selected operations compromise one or more operations add, convolution, concatenation, and element-wise
Gupta [0026] teaches a kernel library with operations including convolutions and any kind of mathematical manipulation.
Regarding claim 5:
The method of claim 1, wherein the convolution-wise operation comprises one or more operations of convolution, deconvolution or transposed convolution, deformable convolution, depth-wise convolution, and grouped convolution, with any stride, dilation, and padding.
Gupta [0033] teaches a convolution-wise operation being a convolution operation.
Regarding claim 6:
A kernel reparameterization built according to claim 1 used in training of the neural network.
Gupta does not teach “used in training of the neural network.”
Ben-dror does teach “used in training of the neural network.” (Fig 9 teaches training the neural network after the kernel layers have been folded or reparametrized.)
Claims 2 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (US 20160335120 A1) in view of Ben-dror et al. (US 20220327386 A1) in view of Zhang et al. (US 20210271973 A1).
Regarding claim 2:
The method of claim 1, wherein the tensor blocks compromise a 1x1 kernel, …, an MxP kernel, an NxN kernel, and an identify kernel, wherein M ≤ N, P ≤ N, and each M, N, and P is a natural number.
Ben-dror Fig 8 with [0090 – 0092] teaches an identity kernel/function or linear layer on a 3x3 sized kernel whose boundary values are 0. [0114] teaches 1 × 1, 3 × 3, and 5 × 5 kernels where M = 3, P = 3, N = 5, M ≤ N, P ≤ N, and M, P, N are all natural numbers.
Gupta over Ben-dror does not teach “a 1xN kernel, an Nx1 kernel”.
Zhang does teach “a 1xN kernel, an Nx1 kernel”. (Zhang [0020] teaches a 1×N vector as a fully connected layer or kernel. N×1 is just the same dimensions of that vector but transposed. The claimed kernel ranges overlap inside ranges disclosed by the prior art, per MPEP 2144.05)
It would have been obvious to for one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gupta over Ben-dror’s teaching of different kernels with Zhang’s teaching of 1 dimensional kernels or vectors since these vectors make up higher dimension kernels. Extra tensor blocks in the form of vectors makes more matrix operations of varying sizes viable, speeds up calculations, and simplifies math processes.
Regarding claim 4:
The method of claim 3, wherein the kernel reparameterization is a linear combination of the selected one or more tensor blocks.
Gupta over Ben-dror does not teach “a linear combination of the selected one or more tensor blocks”.
Zhang does teach “a linear combination of the selected one or more tensor blocks”. ([0008] teaches a network layer being a linear combination of convolution kernels.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gupta over Ben-dror’s teaching of tensor operations with Zhang’s teaching of a linear combination of blocks because using it can reduce the number of complex operations necessary and simplify higher order convolutions, or any other operations within Gupta’s kernel library, into less complex and more efficient computations.
Claims 7, 9, 11, 12, 14, 16, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (US 20160335120 A1) in view of Ben-dror et al. (US 20220327386 A1) in view of Jacob et al. (1712.05877v1)
Regarding claim 7:
A method of performing quantization aware training (QAT) of a neural network, comprising:
identifying a convolution-wise operation kernel in training of the neural network;
Ben-dror [0120] teaches identifying convolution kernels. Fig. 9 block 920 teaches a neural network is trained after the reparametrizing happens.
selecting one or more blocks from tensor blocks and operations;
Rejected for the same reasons as claim 1.
(c) connecting the selected one or more blocks with the selected operations to build a kernel reparameterization, wherein the kernel reparameterization has a dimension same as that of the convolution-wise operation kernel;
Rejected for the same reasons as claim 1.
replacing the convolution-wise operation kernel with the kernel reparameterization;
Gupta [0033] teaches changing a convolution layer kernel with a new or updated operation by reparametrizing.
Gupta in view of Ben-dror doesn’t teach “(d) adding fake quant operator right after the convolution-wise operation;”
Jacob does teach “(d) adding fake quant operator right after the convolution-wise operation;” (Fig 1.a teaches a convolution wise operation. Fig 1.b. teaches simulating quantization by adding weight and activation quantization. Both make up the fake quant operator used for simulating a quantization and the activation quant is applied directly after the convolution.)
It would have been obvious to for one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gupta’s over Ben-dror’s teaching of a kernel reparameterization of convolution operations in a neural network, with Jacob’s teaching of quantizing the reparameterization to reduce memory of the total neural network size while maintaining model accuracy.
(f) … with the kernel reparameterization, wherein the kernel weight is calculated on-the-fly using step (c).
Ben-dror [0036] teaches kernel or node weights which are adjusted while training a neural network or on the fly. [0037] teaches the network is a CNN. In this scenario, the training process includes reparametrizing a convolution operation by convolution folding. [0110] further breaks down what the weights of the network entail.
Gupta in view of Ben-dror doesn’t teach “performing quantization-aware-training of the neural network”
Jacob does teach “performing quantization-aware-training of the neural network” ([Col 1, Lines 1-6] teach quantizing weights and activations of a CNN. Fig 1.1.(b) also teaches training of a convolutional layer with simulated quantization.)
It would have been obvious to for one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gupta over Ben-dror’s teaching of a kernel reparameterization of convolution operations in a neural network and kernel weight calculation, with Jacob’s teaching of quantization-aware-training for a neural network. Reason being, quantizing the convolutions would save memory space and downsize the model making it easier for shipping and deploying.
Claims 8, 10, 13, 15, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Gupta et al. (US 20160335120 A1) in view of Ben-dror et al. (US 20220327386 A1) in view of of Jacob et al. (1712.05877v1) in view of Zhang et al. (US 20210271973 A1).
Regarding claim 8:
The method of claim 7, wherein the tensor blocks compromise a 1x1 kernel, …, an MxP kernel, an NxN kernel, and an identify kernel, wherein M ≤ N, P ≤ N, and each M, N, and P is a natural number.
Ben-dror Fig 8 with [0090 – 0092] teaches an identity kernel/function or linear layer on a 3x3 sized kernel whose boundary values are 0. [0114] teaches 1 × 1, 3 × 3, and 5 × 5 kernels where M = 3, P = 3, N = 5, M ≤ N, P ≤ N, and M, P, N are all natural numbers.
Gupta over Ben-dror over Jacob does not teach “a 1xN kernel, an Nx1 kernel”.
Zhang does teach “a 1xN kernel, an Nx1 kernel”. (Zhang [0020] teaches a 1×N vector as a fully connected layer or kernel. N×1 is just the same dimensions of that vector but transposed. The claimed kernel ranges overlap inside ranges disclosed by the prior art, per MPEP 2144.05)
It would have been obvious to for one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gupta over Ben-dror over Jacob’s teaching of different kernels with Zhang’s teaching of 1 dimensional kernels or vectors since these vectors make up higher dimension kernels. Extra tensor blocks in the form of vectors makes more matrix operations of varying sizes viable, speeds up calculations, and simplifies math processes.
Regarding claim 10:
The method of claim 9, wherein the kernel reparameterization is a linear combination of the selected one or more tensor blocks.
Gupta over Ben-dror over Jacob does not teach “a linear combination of the selected one or more tensor blocks”.
Zhang does teach “a linear combination of the selected one or more tensor blocks”. ([0008] teaches a network layer being a linear combination of convolution kernels.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gupta over Ben-dror over Jacob’s teaching of tensor operations with Zhang’s teaching of a linear combination of blocks because the linear combination is just addition and multiplication of the kernels and their weights. These operations are just use cases of Gupta’s kernel library.
Claims 13 and 15 are system claims with the same limitations as claims 8 and 10 and are rejected for the same reasons.
Claims 18 and 20 are machine claims with the same limitations as claims 8 and 10 and are rejected for the same reasons.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AJAY J KANJOOR whose telephone number is (571)270-0965. The examiner can normally be reached Monday-Friday 8am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AJAY J KANJOOR/Examiner, Art Unit 2142
/HAIMEI JIANG/Primary Examiner, Art Unit 2142
Prosecution Insights