Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an
abstract idea without significantly more.
Regarding claim 1,
Step 1:
Claim 1 recites a machine, one of the four statutory categories of patentable subject matter.
Step 2A, Prong I:
Claim 1 further recites the limitations of:
“... select one or more masks to be used with one or more neural networks ...” This limitation recites a mental process. A person can mentally select one or more masks to be used in a neural network. At a high level, a mask is simply a filter in order to determine whether a value or feature should be selected or ignore. As recited in the Specification at paragraph 100, which discloses a mask layer to select which sets of values corresponding to specific positions and/or tokens to ignore, wherein a human’s mind is capable of performing the similar selection or ignoring process.
Step 2A, Prong II:
Claim 1 recites the following additional elements:
“A processor comprising: one or more circuits to select ...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application.
“... select ... based, at least in part, on one or more hardware resources to use the one or more neural networks” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application. The limitation recites the application of a black-box usage of a neural network and one or more hardware resources in performing the one or more neural networks to select a mask, without reciting any unconventional technique to perform the selection based on hardware resources, or improvement toward a computer element, or improvement toward a machine learning algorithm to generate and select a mask, rather than the application of hardware resources, and neural network.
Step 2B:
When considered individually or in combination, the additional limitations and elements of claim 1 does not amount to significantly more than the judicial exception for the same reasons discussed above as to why the additional limitations do not integrate the abstract idea into a practical application. The additional elements of outlined in Step 2A performing functions as designed simply accomplishes execution of the abstract ideas.
The additional element “A processor comprising: one or more circuits to select ...” is a high-level recitation of generic computer components used as a tool, and does not amount to significantly more than the judicial exception for the same reasons discussed above as to why the additional limitations do not integrate the abstract idea into a practical application.
The additional element “... select ... based, at least in part, on one or more hardware resources to use the one or more neural networks” recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not amount to significantly more than the judicial exception for the same reasons discussed above.
In conclusions from above for the elements considered as a mental process, elements reciting high-level recitation of generic computer components used as a tool, and elements reciting a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f) are carried over and do not provide significantly more than the abstract idea. Looking at the limitations in combination and the claims as a whole does not change this conclusion and the claim is ineligible.
Therefore, additional limitations of claim 1 do not amount to significantly more than the judicial exception.
Thus, claim 1 recites abstract ideas with additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Therefore, claim 1 is not patent eligible.
Regarding claim 2 depends on claim 1, thus the rejection of claim 1 is incorporated.
Claim 2 recites the element:
“The processor of claim 1, wherein the one or more circuits ...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception.
“... select the one or more masks ...” This element recites a mental process, because a person can mentally select one or more masks.
“select ... based, at least in part, on one or more metrics generated by performing variations of the one or more neural networks using the one or more hardware resources” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application of performing the one or more neural network using the one or more hardware resources and obtain one or more performance metrics, without reciting the technical detail of how the neural network is performed in an unconventional manner, or improvement toward a computer element.
Thus, claim 2 recites abstract ideas with additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 3 depends on claim 1, thus the rejection of claim 1 is incorporated.
Claim 3 recites the element:
“The processor of claim 1, wherein the one or more circuits ...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception.
“... select the one or more masks to be used with the one or more neural networks to modify one or more data values of the one or more neural networks” This element recites a mental process, because a person can mentally select one or more masks to be used in a neural network. A person can manually determine a mask value for replacing for another value, thereby modify that value. Such replacing and modification can manually and mentally be performed by a person.
Thus, claim 3 recites abstract ideas with additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 4 depends on claim 1, thus the rejection of claim 1 is incorporated.
Claim 4 recites the element:
“The processor of claim 1, wherein the one or more circuits ...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception.
“... prune one or more layers of the one or more neural networks using the one or more masks” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application of using a mask to prune one or more layers of a neural network, without reciting specific technical step to perform the pruning by applying the mask in an unconventional manner or specific technique that demonstrate how a mask can be used to prune a neural network layer.
Thus, claim 4 recites additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 5 depends on claim 1, thus the rejection of claim 1 is incorporated.
Claim 5 recites the element:
“The processor of claim 1, wherein the one or more circuits ...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception.
“... select the one or more masks based, at least in part, on a list that correlates one or more metrics with one or more optimization techniques and one or more parameters of the one or more hardware resources.” This element recites a mental process, because a person can mentally select a mask based on a list that correlates one or more metrics with one or more optimization techniques, and or more parameters of the one or more hardware resources. Such evaluating of a list that contain one or more metrics with one or more optimization techniques as well as one or more parameters can be mentally or manually performed by a human’s mind.
Thus, claim 5 recites abstract ideas with additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 6 depends on claim 1, thus the rejection of claim 1 is incorporated.
Claim 6 recites the element:
“The processor of claim 1, wherein the one or more circuits ...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception.
“... select a mask from the selected one or more masks to be used with the one or more neural networks based, at least in part, on an identification of a hardware resource” This element recites a mental process, because a person can mentally select a mask based at least in part, on an identification of a hardware resource. A person can mentally or manually check and determine the hardware resource to implement/processing a neural network.
Thus, claim 6 recites abstract ideas with additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 7 depends on claim 1, thus the rejection of claim 1 is incorporated.
Claim 7 recites the element:
“The processor of claim 1, wherein the one or more circuits ...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception.
“... modify one or more parameters to be used with the one or more neural networks based, at least in part, on one or more dense versions of the one or more neural networks” This element recites a mental process, because a person can mentally or manually modify one or more parameters to be used with the one or more neural networks based on the version of the network. Since parameter can be a value, such modification to a value can be performed mentally or manually by a person.
Thus, claim 7 recites abstract ideas with additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 8 which recites a system, one of the four statutory categories of patentable subject matter. The applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps.
Regarding claim 9 depends on claim 8, thus the rejection of claim 8 is incorporated. The applicant is further directed to the rejection of claim 2 above, because the claim recites similar limitations and processing steps.
Regarding claim 10 depends on claim 8, thus the rejection of claim 8 is incorporated.
Claim 10 recites the element:
“The system of claim 8, wherein the one or more processors...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception.
“... quantize one or more values of the one or more neural networks based, at least in part, on the one or more masks” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application of quantization technique based on the masks without reciting specific technical step, unconventional quantization process, or improvement to machine learning algorithm or computer element.
Thus, claim 10 recites additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 11 depends on claim 8, thus the rejection of claim 8 is incorporated.
Claim 11 recites the element:
“The system of claim 8, wherein the one or more processors...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception
“... generate one or more sparse neural networks using the selected one or more masks with the one or more neural networks” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application of using a mask to generate the one or more sparse neural network without reciting specific technical step, unconventional method to generate sparse neural network, or improvement to machine learning algorithm or computer element.
Thus, claim 11 recites additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 12 depends on claim 8, thus the rejection of claim 8 is incorporated.
Claim 12 recites the element:
“The system of claim 8, wherein the one or more processors...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception
“... select the one or more masks based, at least in part, on performance metrics generated by the one or more hardware resources performing the one or more neural networks that exceed one or more threshold performance metrics.” This element recites a mental process, because a person can mentally or manually perform the mental evaluation of a performance metrics after using some hardware to perform a neural network. Such determining whether the performance exceed one or more threshold can be mentally performed in a human’s mind.
Thus, claim 12 recites abstract ideas with additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 13 depends on claim 8, thus the rejection of claim 8 is incorporated. The applicant is further directed to the rejection of claim 5 above, because the claim recites similar limitations and processing steps.
Regarding claim 14 depends on claim 8, thus the rejection of claim 8 is incorporated.
Claim 14 recites the element:
“The system of claim 8, wherein the one or more processors...” This additional element is a high-level recitation of generic computer components used as a tool, and does not provide integration into a practical application, and does not amount to significantly more than the judicial exception
“... train the one or more neural networks using the one or more selected masks” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application training a neural network using a mask, without providing specific implementation detail on how the training is perform, or an unconventional training technique, or improvement to machine learning algorithm or computer element.
Thus, claim 14 recites additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 15 which recites a method, one of the four statutory categories of patentable subject matter. The applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps.
Regarding claim 16 depends on claim 15, thus the rejection of claim 15 is incorporated.
Claim 16 recites the element:
“The method of claim 15, wherein the one or more masks is identified based, at least in part, on performance metrics generated by using the one or more hardware resources performing the one or more neural network models modified with a quantization of one or more values.” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application of quantization technique, and further application of hardware resource to carry neural network processing and obtain performance metric, without reciting specific technical step of quantization and neural network training, unconventional quantization process and neural network training, or improvement to machine learning algorithm or computer element.
Thus, claim 16 recites additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 17 depends on claim 15, thus the rejection of claim 15 is incorporated. The applicant is further directed to the rejection of claim 5 above, because the claim recites similar limitations and processing steps.
Regarding claim 18 depends on claim 15, thus the rejection of claim 15 is incorporated.
Claim 18 recites the element:
“The method of claim 15, further comprising generating one or more sparse neural networks using the one or more masks to reduce one or more layers of the one or more neural networks” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application of the one or more masks to generate a one or more sparse neural networks with reduced layers, without providing the technical detail in how the mask can be applied to reduce the layer of the one or more neural networks in an unconventional manner, or improvement to a neural network generating algorithm based on mask or computer element.
Thus, claim 18 recites additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 19 depends on claim 15, thus the rejection of claim 15 is incorporated.
Claim 19 recites the element:
“The method of claim 15, wherein the one or more neural networks are to be performed by the one or more hardware resources to exceed one or more threshold performance metrics that indicate accuracy and one or more threshold values that indicate inferencing speed” This additional element recites an additional element of a mere instruction to apply an exception with a recitation of the words "apply it" (or an equivalent) as identified in MPEP 2106.05(f), and does not provide integration into a practical application or amount to significantly more than the judicial exception. The limitation recites the black-box application of using hardware to perform neural network and obtain performance according to metric such as accuracy and inferencing speed. The claim does not provide technical detail to generate the neural network in an unconventional manner or improvement toward a machine learning neural network or computer element.
Thus, claim 19 recites additional elements rendered at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Regarding claim 20 depends on claim 15, thus the rejection of claim 15 is incorporated.
Claim 20 recites the element:
“The method of claim 15, wherein the one or more masks are one or more tensors that indicate which nonzero values of the one or more neural networks should be set to zero” This element recites a mental process, because a person can mentally or manually indicate a mask or value to determine whether a value should be set to zero or not. Such determination of whether a value should be set to zero can be performed manually or mentally by a human’s mind.
Thus, claim 20 recites abstract ideas at a high level of generality resulting in claims that do not integrate the abstract idea into a practical application or amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 6-9, 11-15, 17-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Xu et.al (US 20190362235 A1)
Regarding claim 1,
Xu teaches the limitation “A processor comprising: one or more circuits to select one or more masks to be used with one or more neural networks based, at least in part, on one or more hardware resources to use the one or more neural networks” (paragraph 40 “generating a pruned, sparse version of a neural network utilizing a hybrid pruning approach ... The resulting sparse neural network model may also result in decreased computation and power demands with little to no loss in accuracy, thereby enabling sophisticated, modern neural networks to be deployed on resource-constrained devices, such as always-on security cameras, IoT devices, drones, among other examples”, paragraph 46 “a mask may be generated 525 based on this pruning percentage and the sorting 520. The channels may then be pruned 530 according to the mask ... A pruned version of the neural network may then be likewise generated that includes the pruned version of this layer”, and paragraph 62 “Indeed, computing devices, processors, and other logic and circuitry of the systems described herein may incorporate all or a portion of the functionality and supporting software and/or hardware circuitry to implement such functionality” Xu discloses a method and system for pruning neural network for resource-constrained devices, wherein the pruning is performed based on generating a mask. The method and system of pruning/masking may be performed by one or more hardware devices with processors and circuitries. The pruning comprises generating a mask to prune channels corresponding to neural network layers, thereby generating pruned neural network to be deployed on resource-constrained devices (e.g., always-on security cameras, IoT devices, drones), which teaches the selection of one or more mask to be used with one or more neural networks based, at least in part, on one or more hardware resources to use the one or more neural networks as claimed, because in Xu, the pruning and masking is performed based on a demand to deploy a pruned neural network for resource-constrained devices without detrimentally sacrificing accuracy, thus a mask is generated and selected for pruning the neural network, based on a demand that a resource-constrained device need a pruned neural network.)
Regarding claim 2 depends on claim 1, thus the rejection of claim 1 is incorporated.
Xu teaches the limitation “The processor of claim 1, wherein the one or more circuits are to select the one or more masks based, at least in part, on one or more metrics generated by performing variations of the one or more neural networks using the one or more hardware resources” (paragraph 46 “The channels may then be pruned 530 according to the mask to generate a pruned version of the layer ... The pruned version of the neural network may then be caused to be implemented on a computing platform and tested 535 against a set of test input data to determine what affect this initial pruning of the particular layer has on the overall accuracy of the neural network model. If the pruned version of the neural network has an accuracy that is within an acceptable range or above an acceptable threshold set for the pruning” Xu discloses the pruned version of the neural network based on generating a mask is tested against test input data to determine the accuracy of the pruned neural network based on an acceptable range/threshold, which teaches the selection of the one or more masks based, at least in part, on one or more metrics generated by performing variations of the one or more neural networks as claimed, wherein the accuracy corresponds to the metrics.)
Regarding claim 3 depends on claim 1, thus the rejection of claim 1 is incorporated.
Xu teaches or at least suggests the limitation “The processor of claim 1, wherein the one or more circuits are to select the one or more masks to be used with the one or more neural networks to modify one or more data values of the one or more neural networks” (paragraph 55 “In one example, statistics-aware weight pruning may be performed for fine-grained weight pruning. For instance, a layer-wise weight threshold may be computed based on the statistical distribution of full dense weights in each channel-pruned layer and weight pruning may be performed to mask out those weights that are less than the corresponding layer-specific threshold”, and paragraph 56 “In one example implementations, a layer-wise mask may be defined to represent the binary mask governing which weights to prune during any given training iteration”. Xu discloses performing weight pruning by implementing a layer-wise mask to represent a binary mask governing which weights to prune to mask out those weights and alter train the neural network with pruned weights, thereby teaches the selection via generating the one or more masks to be used with the one or more neural networks to modify one or more data values of the one or more neural networks as claimed., wherein the weight parameters correspond to one or more data values, and the weight pruning corresponds to the modification of the one or more data value.)
Regarding claim 4 depends on claim 1, thus the rejection of claim 1 is incorporated.
Xu teaches the limitation “The processor of claim 1, wherein the one or more circuits are to prune one or more layers of the one or more neural networks using the one or more masks” (paragraph 46 “a mask may be generated 525 based on this pruning percentage and the sorting 520. The channels may then be pruned 530 according to the mask to generate a pruned version of the layer. A pruned version of the neural network may then be likewise generated that includes the pruned version of this layer” Xu discloses using a mask to prune the channels that corresponding to a neural network layer, which corresponds to the pruning one or more layers of the one or more neural networks using the one or more masks, as claimed.)
Regarding claim 6 depends on claim 1, thus the rejection of claim 1 is incorporated.
Xu teaches the limitation “The processor of claim 1, wherein the one or more circuits are to select a mask from the selected one or more masks to be used with the one or more neural networks based, at least in part, on an identification of a hardware resource” (paragraph 40 “generating a pruned, sparse version of a neural network utilizing a hybrid pruning approach ... The resulting sparse neural network model may also result in decreased computation and power demands with little to no loss in accuracy, thereby enabling sophisticated, modern neural networks to be deployed on resource-constrained devices, such as always-on security cameras, IoT devices, drones, among other examples”, and paragraph 46 “a mask may be generated 525 based on this pruning percentage and the sorting 520. The channels may then be pruned 530 according to the mask ... A pruned version of the neural network may then be likewise generated that includes the pruned version of this layer” Xu discloses generating a pruned neural network for resource-constrained devices, such as always-on security cameras, IoT devices, drones, thereby teaches an identification of a hardware resource, as claimed. Furthermore, since the pruning comprises generating a mask and test the pruned neural network for training on a test data input, thereby teaches or at least suggests the selection of a mask to be used with the one or more neural network of the hardware resource, as claimed.)
Regarding claim 7 depends on claim 1, thus the rejection of claim 1 is incorporated.
Xu teaches the limitation “The processor of claim 1, wherein the one or more circuits are to modify one or more parameters to be used with the one or more neural networks based, at least in part, on one or more dense versions of the one or more neural networks” (paragraph 52 “For instance, beginning with an initial, unpruned or dense version of a neural network model 505, ... a pruned version of the neural network may be formed from the determined layers and may be fine-tuned and retrained”, paragraph 53 “In the case of hybrid pruning, an additional fine-grained pruning step may be performed to further reduce the size and complexity of the model following the completion of coarse-grained channel pruning ... In one example, statistics-aware weight pruning may be performed for fine-grained weight pruning ... weight pruning may be performed to mask out those weights” and paragraph 54 “After each training iteration, the weight parameters may be updated 580, and the next iteration of training performed with corresponding pruning and splicing 575, and so on until the network stabilizes” Xu discloses the training of the dense version of the neural network using hybrid pruning, which comprise the step of using a mask to prune weight parameter and further retrain the pruned neural network, thereby teaches modifying one or more parameters to be used with the one or more neural networks based, at least in part, on one or more dense versions of the one or more neural networks, as claimed.)
Regarding claim 8, the applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps.
Regarding claim 9 depends on claim 8, thus the rejection of claim 8 is incorporated. The applicant is further directed to the rejection of claim 2 above, because the claim recites similar limitations and processing steps.
Regarding claim 11 depends on claim 8, thus the rejection of claim 8 is incorporated.
Xu teaches the limitation “The system of claim 8, wherein the one or more processors are to generate one or more sparse neural networks using the selected one or more masks with the one or more neural networks” (paragraph 40 “In some implementations, a computing system (e.g., 105) may be equipped with logic implemented in hardware and/or software that is capable of generating a pruned, sparse version of a neural network utilizing a hybrid pruning approach, which combines both coarse-grained channel pruning and fine-grained weight pruning to reduce the overall model size. The resulting sparse neural network model ...” Xu discloses using the hybrid pruning approach to generate sparse version of a neural network, wherein the pruning comprises generating one or more masks (layer-wise mask/binary mask), thereby teaches the claimed process of generating one or more sparse neural networks using the selected one or more masks with the one or more neural networks.)
Regarding claim 12 depends on claim 8, thus the rejection of claim 8 is incorporated.
Xu teaches the limitation “The system of claim 8, wherein the one or more processors are to select the one or more masks based, at least in part, on performance metrics generated by the one or more hardware resources performing the one or more neural networks that exceed one or more threshold performance metrics.” (paragraph 127 “Example 1 is a machine accessible storage medium having instructions stored thereon, where the instructions when executed on a machine, cause the machine to: access data including a definition of a neural network, ... providing input data to a thinned version of the neural network in a test, where the thinned version of the neural network includes the thinned version of the layer; determining accuracy of the thinned version of the neural network based on an output of the neural network in the test; and adopting the thinned version of the layer to generate the pruned version of the layer based on the accuracy of the thinned version of the neural network exceeding a threshold accuracy value” Xu discloses a machine execute a pruned version of the neural network (the pruning process is performed by generating a mask) is tested against test input data to determine if the accuracy of the pruned neural network exceed a threshold accuracy value, which teaches the selection of the one or more masks based, at least in part, on one or more metrics generated by the one or more hardware resources performing the one or more neural networks that exceed one or more threshold performance metrics.)
Regarding claim 13 depends on claim 8, thus the rejection of claim 8 is incorporated. the applicant is further directed to the rejection of claim 5 above, because the claim recites similar limitations and processing steps.
Regarding claim 14 depends on claim 8, thus the rejection of claim 8 is incorporated.
Xu teaches the limitation “The system of claim 8, wherein the one or more circuits are further to train the one or more neural networks using the one or more selected masks” (paragraph 56 “In one example implementations, a layer-wise mask may be defined to represent the binary mask governing which weights to prune during any given training iteration” Xu discloses the training of a neural network is performed while a mask is generated to perform the pruning of weight parameter, thereby teaches the process of training the one or more neural networks using the one or more selected masks, as claimed.)
Regarding claim 15, the applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps.
Regarding claim 17 depends on claim 15, thus the rejection of claim 15 is incorporated.
Xu teaches the limitation “The method of claim 15, wherein the one or more masks is to be used with the one or more neural networks to prune one or more portions of the one or more neural networks” (paragraph 56 “In one example implementations, a layer-wise mask may be defined to represent the binary mask governing which weights to prune during any given training iteration” Xu discloses using the binary mask to govern which weights to prune during training of a neural network, thereby teaches the one or more masks is to be used with the one or more neural networks to prune one or more portions of the one or more neural networks, as claimed.)
Regarding claim 18 depends on claim 15, thus the rejection of claim 15 is incorporated.
Xu teaches the limitation “The method of claim 15, further comprising generating one or more sparse neural networks using the one or more masks to reduce one or more layers of the one or more neural networks.” (paragraph 38 “As illustrated in FIG. 3, hybrid pruning may be performed on the original version of the neural network 305 to thin one or more of the layers of the model ... In this example hybrid pruning 330 may be performed, in which coarse-grained and fine-grained pruning are performed on layers 310, 315, 320, to generate thinned layers 310′, 315′, 320′. These thinned layers may replace the original, dense versions of the layers 310, 315, 320 to form a thinned, or pruned, version of the neural network 305′. This thinned, or sparse, neural network model 305′ may be dramatically smaller in size, making the model 305′ well-suited for use”, and paragraph 55 “In one example, after course-grained channel pruning is performed to reduce the size of the layer” Xu discloses the hybrid pruning process, in which an original neural network is pruned (the pruning process is performed by generating a mask) to generate the thinned layer or reduced size of the layer of the neural network, thereby teaches the process of generating one or more sparse neural networks using the one or more masks to reduce one or more layers of the one or more neural networks.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5, 10, 16, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et.al (US 20190362235 A1) in view of Aytekin et.al (US 20220164652 A1)
Regarding claim 5 depends on claim 1, thus the rejection of claim 1 is incorporated.
Xu teaches or at least suggest the limitation “The processor of claim 1, wherein the one or more circuits are to select the one or more masks based, at least in part, on a list that correlates one or more metrics with one or more optimization techniques and one or more parameters of the one or more hardware resources” (paragraph 38 “hybrid pruning 330 may be performed, in which coarse-grained and fine-grained pruning are performed on layers 310, 315, 320, to generate thinned layers′”, paragraph 44 “Sensitivity based channel pruning may seek to identify the upper bound in the percentage of the number of intact output channels in each layer with acceptable accuracy loss ... To define how much accuracy loss is too much, an accuracy tolerance may be defined for the neural network”, paragraph 45 “An accuracy threshold may be defined that is specific to the neural network or neural networks of a particular type, and this accuracy threshold (at 510) may be utilized during sensitivity testing performed with the coarse-grained pruning stage of a hybrid pruning”, paragraph 54 “Weight pruning may be performed when a given weight's value falls below a threshold value, thereby causing the value to be reduced to zero. However, if the weight value evolves to be above the threshold value, the weight may be spliced, or restored”, and paragraph 49 “In some implementations, the number of preserved (i.e., unpruned) channels may be rounded up or down to a number corresponding to the architecture of the system that is to use and perform calculations based on the neural network model. For instance, the preserved channels may be selected to be a number corresponding to the number of multiply-accumulate (MAC) circuits, the number of cores, or another number corresponding to a hardware architecture” Xu discloses one or more metrics corresponding to accuracy-related metrics, including accuracy loss, accuracy tolerance, and an accuracy threshold used to determine whether pruning causes unacceptable accuracy. Xu further discloses one or more optimization techniques corresponding to hybrid pruning, coarse-grained pruning, and fine-grained pruning used to determine pruning percentages and generate masks. Each pruning technique may correspond with an accuracy-related metrics, and a person of ordinary skill in the art would have found it obvious to organize this associated information in a list, table, or other data structure that correlates each accuracy metrics with each pruning techniques. Such organization would allow the system to efficiently compare available pruning/mask configurations in view of the hybrid pruning technique and select a mask for a particular neural-network layer while satisfying the desired accuracy threshold/tolerance. Thus, Xu at least suggests select the one or more masks based, at least in part, on a list that correlates one or more metrics with one or more optimization techniques, as claimed. Xu also discloses one or more parameters of the hardware resources corresponding to the number of MAC circuits, number of cores, or other hardware-architecture parameters used to select preserved or unpruned channels in which a mask may be further generated, which teaches or at least suggests one or more parameters of the one or more hardware resources, as claimed.)
Regarding claim 10 depends on claim 8, thus the rejection of claim 8 is incorporated.
Xu does not teach the limitation “The system of claim 8, wherein the one or more processors quantize one or more values of the one or more neural networks based, at least in part, on the one or more masks”. However, Aytekin teaches or at least suggest this limitation (paragraph 73 “According to an embodiment, the weight tensor is quantized. In other words, the weights or biases of the neural network layers are quantized. Quantization causes the tensor to be more easily compressed”, paragraph 75 “Quantization may be approximated by introducing additive noise to the weight tensor during training”, and paragraph 89 “After the pruning and quantization one can code the zero elements in a binary mask that indicates which element is zero and which is not.” Aytekin discloses quantization of the weights or biases of the neural network, and further teaches than quantization may be approximated by introducing additive noise during training of a neural network, thereby teaches or at least suggest the quantizing one or more values of the one or more neural networks, as claimed. Furthermore, because Xu already teaches the weights are represented by the binary masks above, a person of ordinary skill in the art would have been motivated to apply Aytekin’s quantization to Xu’s mask-based punning process such that the quantized neural network values are processed based at least in part on the binary mask of the weights, since the non-zero weights may still consume memory/computation because they are high-precision values, quantization may further compress those remaining values and reduces storage/computation, thereby teaches or at least suggests quantizing one or more values of the one or more neural networks based, at least in part, on the one or more masks, as claimed.)
Before the effective filing date, it would have been obvious to one of ordinary skill in the art to combine the teaching of hybrid neural network pruning by generating a mask by Xu with the teaching of quantizing values of neural network by Aytekin. The motivation to do so is referred to in Aytekin’s disclosure (paragraph 73 “Thus, quantization of weights or biases is an important aspect to be considered for compression”, paragraph 74 “Quantization may comprise approximating the quantization during training such that the neural network would learn to be robust to quantization after training. This way the possibly decreased performance of the neural network caused by direct quantization of the weights after training, based on a uniform or non-uniform quantization, may be avoided. The performance may decrease, since the neural network is not robust to direct quantization after training” Aytekin discloses the importance of quantization of weights or biases values of neural network in compression problem. Therefore, a person of ordinary skill in the art would have been motivated to apply Aytekin’s quantization technique to Xu’s hybrid neural-network pruning process because doing so would further reduce the size/storage requirement of the neural network and improve compression of values within the pruned neural-network model. Aytekin further discloses that performing/approximating quantization during training allows the neural network to become robust to quantization after training, thereby reducing performance degradation caused by direct quantization after training. Accordingly, the combination would have predictably resulted in a compressed and efficient neural network that maintains acceptable performance, which is consistent with Xu’s goal of generating a pruned neural network using masks to reduce computational/resource requirements.)
Regarding claim 16 depends on claim 15, thus the rejection of claim 15 is incorporated.
Xu in view of Aytekin teaches or at least suggest the limitation “The method of claim 15, wherein the one or more masks is identified based, at least in part, on performance metrics generated by using the one or more hardware resources performing the one or more neural network models modified with a quantization of one or more values” (Aytekin at paragraph 74 “Quantization may comprise approximating the quantization during training such that the neural network would learn to be robust to quantization after training. This way the possibly decreased performance of the neural network caused by direct quantization of the weights after training, based on a uniform or non-uniform quantization, may be avoided. The performance may decrease, since the neural network is not robust to direct quantization after training”, and paragraph 89 “After the pruning and quantization one can code the zero elements in a binary mask that indicates which element is zero and which is not” Aytekin discloses that quantization affects neural network performance and teaches performing the quantization during the training of the neural network to avoid performance degradation. Aytekin further discloses using a binary mask after pruning and quantization. Therefore, when Aytekin’s quantization and masking process is applied to Xu’s mask-based pruning process, a person of ordinary skill in the art would have found it obvious to further identify/select a binary mask based at least in part on determine if the pruned neural network is acceptable by comparison with an accuracy performance threshold obtained from Xu’s devices executing/testing pruned neural network trained using Aytekin’s quantized weight values, thereby teaches or at least suggests the claimed process.)
The motivation to combine the teaching of Xu with Aytekin is similar to the motivation as recited in claim 10 above.
Regarding claim 19 depends on claim 15, thus the rejection of claim 15 is incorporated.
Xu teaches or at least suggest the limitation “The method of claim 15, wherein the one or more neural networks are to be performed by the one or more hardware resources to exceed one or more threshold performance metrics that indicate accuracy and one or more threshold values that indicate inferencing speed” (paragraph 54 “An accuracy threshold may be defined that is specific to the neural network or neural networks of a particular type, and this accuracy threshold (at 510) may be utilized during sensitivity testing performed with the coarse-grained pruning stage of a hybrid pruning”, and paragraph 51 “For instance, it should be appreciated that each version of the neural network model that is to be iteratively tested in the sensitivity test (with a respective pruned version of a single one of the model layers) does not need to be retrained ... Such tests may thereby be completed quickly with only modest computing resources (e.g., 8.86 minutes on 1-CPU Intel Core i7-6850K CPU and 3.38 seconds on GPU GTX-1080-Ti Pascal for sensitivity tests on ResNet50 on ImageNet) ... Sensitivity testing can be performed during inference time” Xu discloses an accuracy threshold may be defined that is specific to the neural network, which corresponds to the one or more threshold performance metrics that indicate accuracy, as claimed. Xu also discloses sensitivity testing may be performed during inference time and completed using computing resources. Because the sensitivity testing evaluates the pruned neural network during inference-time operation and provide the time as in the example, a person of ordinary skill in the art would have understood that the time/speed required to perform the inference-time testing is an inference-speed performance metric. It would have been obvious to compare such inference-time/performance to a threshold value to determine whether the pruned neural network satisfies a desired inferencing speed requirement.)
Regarding claim 20 depends on claim 15, thus the rejection of claim 15 is incorporated.
Xu teaches or at least suggests the limitation “The method of claim 15, wherein the one or more masks are one or more tensors that indicate which nonzero values of the one or more neural networks should be set to zero” (paragraph 56 “In one example implementations, a layer-wise mask may be defined to represent the binary mask governing which weights to prune during any given training iteration ... n this example, fine-grained weight pruning may be performed by sparsifying weights (preserving or forcing to zero the value of the weight based on a comparison with a threshold weight value) in the forward pass according to the mask” Xu discloses a layer-wise mask may be defined to represent the binary mask governing which weights to prune, and that fine-grained weight pruning may be performed by preserving or forcing to zero the value of the weight according to the binary mask. Thus, xu teaches or at least suggests the one or more masks that indicate which nonzero values of the one or more neural networks should be set to zero, as claimed. A person of ordinary skill in the art would have understood or found it obvious that the mask would be implemented as a tensor/array having entries corresponding to the weights of the neural-network layer. Therefore, Xu teaches or at least suggests the one or more masks are one or more tensors that indicate which nonzero values of the one or more neural networks should be set to zero, as claimed.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY TU DIEP whose telephone number is (703)756-1738. The examiner can normally be reached M-F 8-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY T DIEP/ Examiner, Art Unit 2123
/ALEXEY SHMATOV/ Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights