DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the application and claims filed 8/22/2023. Claims 1-25 are pending and have been examined. Claims 1-25 are rejected.
Information Disclosure Statement
Acknowledgment is made of the information disclosure statements filed 8/22/2023, 7/02/2024, and 4/29/2026, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis below of the claims’ subject matter eligibility follows the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (January 7, 2019) (“2019 PEG”) and the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence, 89 Fed. Reg. 58128-58138 (July 17, 2024) (“2024 AI SME Update”).
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself.
Regarding independent claim 1, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 1 is directed to a method corresponding to a process, one of the statutory categories.
Step 2A Prong 1: The claim recites
“selecting a layer from the plurality of layers in the neural network, the selected layer in the neural network generating a tensor based on the dataset;
pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold;
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor;
determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold;
modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold.”
The “selecting a layer from the plurality of layers” limitation, as drafted, under its broadest reasonable interpretation (BRI), covers concepts performed in the human mind including an observation, evaluation, judgement, or opinion to select a layer in the generically-recited neural network in order to generate/populate a tensor based on observed data in “the dataset.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “pruning the tensor … by modifying an absolute value of an activation in the tensor to zero” limitation covers a mathematical concept of modifying an absolute value of an activation in the tensor. Such modification is discussed in paragraph 216 of the specification, further providing evidence that the claimed pruning the tensor by modifying an absolute value is itself a mathematical concept.
The “determining an accuracy of the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine an observed accuracy of the generically-recited neural network (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining a second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “a second activation threshold based on the first activation threshold and the accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “modifying the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) based on observed data in the dataset.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The claim recites:
“A method for modifying a neural network, comprising:
inputting a dataset into the neural network, the neural network comprising a plurality of layers;”
The judicial exceptions are not integrated into a practical application. In particular, the claim recites the “modifying a neural network” limitation which amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer or merely uses a computer as a tool to perform an abstract idea (i.e., generic computer components – a generically-recited neural network, performing generic computer functions which does not integrate a judicial exception into a practical application. See MPEP 2106.05(f).
The “inputting a dataset into the neural network” limitation describes data gathering with a dataset as the data. This inputting a dataset limitation can be characterized as insignificant extra-solution activity (i.e., data gathering). See MPEP 2106.05(g).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The “modifying a neural network” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element represents mere instructions to apply an exception (i.e., the additional element recites a method for modifying a neural network for applying the abstract ideas). Mere instructions to alter the neural network do not amount to significantly more than the judicial exception. As noted above, merely asserting that a judicial exception is to be carried out on a generic neural network cannot provide significantly more than the judicial exception. Mere instructions to apply an exception cannot provide an inventive concept.
The “inputting a dataset into the neural network” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, recitations of “inputting a dataset into the neural network” are well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
This claim is not patent eligible.
Regarding claim 2, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 2 is directed to a method as depending from claim 1, thus the analysis for patent eligibility of claim 1 is incorporated herein.
Step 2A Prong 1: The claim recites:
“determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network;
determine whether the accuracy loss exceeds a threshold; and
in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold.”
The “determining the second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “an accuracy loss caused by pruning the tensor based on the accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determine … the accuracy loss” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “whether the accuracy loss exceeds a threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining that the accuracy loss is lower than the threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold by increasing the first activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 3, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 3 is directed to a method as depending from claim 2, thus the analysis for patent eligibility of claim 2 is incorporated herein.
Step 2A Prong 1: The claim recites:
“in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold.”
The “determining that the accuracy loss exceeds the threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold by decreasing the first activation threshold”(corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 4, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 4 is directed to a method as depending from claim 1, thus the analysis for patent eligibilities/eligibility of claim 1 and is incorporated herein.
Step 2A Prong 1: The claim recites:
“determining the first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.”
The “determining the first activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the first activation threshold based on a third activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 5, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 5 is directed to a method as depending from claim 1, thus the analysis for patent eligibility of claim 1 is incorporated herein.
Step 2A Prong 1: The claim recites:
“pruning the additional tensor based on the third activation threshold;
determining an additional accuracy of the neural network based on an additional output generated by the neural network based on the pruned additional tensor; and
determining the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.”
The “pruning the additional tensor” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “which tensor to prune.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining an additional accuracy of the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “an additional accuracy of the neural network based on an additional output generated by the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining the first activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The claim recites:
“inputting an additional dataset into the neural network, the selected layer in the neural network generating an additional tensor based on the additional dataset.”
The “inputting an additional dataset into the neural network” limitation describes data gathering with “an additional dataset” as the data. This inputting an additional dataset into the neural network limitation can be characterized as insignificant extra-solution activity (i.e., data gathering). See MPEP 2106.05(g).
Accordingly, these/this additional element(s) do/does not integrate the abstract idea into a practical application because <they do not/ it does not> impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The “inputting an additional dataset into the neural network” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, recitations of “inputting an additional dataset into the neural network, the selected layer in the neural network generating an additional tensor” are well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
This claim is not patent eligible.
Regarding claim 6, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 6 is directed to a method as depending from claim 1, thus the analysis for patent eligibility of claim 1 is incorporated herein.
Step 2A Prong 1: The claim recites:
“selecting another layer in the neural network; and
modifying the neural network by adding another activation pruning operation to the another layer, the another activation pruning operation to prune one or more tensors to be generated by the another layer based on another activation threshold.”
The “selecting another layer in the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine the layer to prune in the neural network (corresponding to mental processes which can be done mentally or by pen and paper).
The “modifying the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “based on another activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 7, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 7 is directed to a method as depending from claim 1, thus the analysis for patent eligibility of claim 1 is incorporated herein.
Step 2A Prong 1: The claim recites:
“selecting the layer based on an amount of internal parameters of the layer, an amount of computations in the layer, a type of the layer, or some combination thereof.”
The “selecting the layer” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the layer based on an amount of internal parameters of the layer.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 8, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 8 is directed to a method as depending from claim 1, thus the analysis for patent eligibility of claim 1 is incorporated herein.
Step 2A Prong 1: The claim recites:
“further modifying the neural network by adding a weight pruning operation to the selected layer, the activation pruning operation to prune a kernel of the selected the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero,
wherein the absolute value of the weight is lower than the weight threshold.”
The “modifying the neural network” limitation added by this claim covers a mathematical concept of “modifying an absolute value of a weight in the kernel to zero”. Such modification by modifying an absolute value of a weight in the kernel to zero is discussed in paragraph 223 of the specification, further providing evidence that the claimed pruning operation by modifying an absolute value of a weight in the kernel to zero is itself a mathematical concept.
The “absolute value of the weight” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “if the weight is lower than the weight threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 9, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 9 is directed to a method as depending from claim 8, thus the analysis for patent eligibility of claim 8 is incorporated herein.
Step 2A Prong 1: The claim recites:
“to determine the absolute value of the weight”
The “determine the absolute value of the weight” limitation added by this claim covers a mathematical concept of determining the absolute value of the weight. Such determination is discussed of in paragraph 224 of the specification, further providing evidence that the claimed neural network training by determining the absolute value of the weight is itself a mathematical concept.
Step 2A Prong 2: The claim recites:
“wherein the neural network has been trained”
The judicial exceptions are not integrated into a practical application. In particular, the claim recites the generically-recited “neural network has been trained” limitation which amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer or merely uses a computer as a tool to perform an abstract idea (i.e., generic computer components – generic neural network performing generic computer functions) which does not integrate a judicial exception into a practical application. See MPEP 2106.05(f).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The “neural network has been trained” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element represents mere instructions to apply an exception (i.e., the additional element recites “wherein the neural network has been trained” for applying the abstract ideas). Mere instructions to train the neural network do not amount to significantly more than the judicial exception. As noted above, merely asserting that a judicial exception is to be carried out on a generic neural network cannot provide significantly more than the judicial exception. Mere instructions to apply an exception cannot provide an inventive concept.
Regarding claim 10, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 10 is directed to a method as depending from claim 8, thus the analysis for patent eligibility of claim 8 is incorporated herein.
Step 2A Prong 1: The claim recites:
“determining the second activation threshold after adding the weight pruning operation to the layer.”
The “determining the second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding independent claim 11, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 11 is directed to a non-transitory computer-readable media corresponding to an article of manufacture, one of the statutory categories.
Step 2A Prong 1: The claim recites
“selecting a layer from the plurality of layers in the neural network, the selected layer in the neural network generating a tensor based on the dataset;
pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold;
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor;
determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold;
modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold.”
The “selecting a layer from the plurality of layers” limitation, as drafted, under its “broadest reasonable interpretation (BRI)”, covers concepts performed in the human mind including an observation, evaluation, judgement, or opinion to select a layer in the generically-recited neural network in order to generate/populate a tensor based on observed data in “the dataset.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “pruning the tensor … by modifying an absolute value of an activation in the tensor to zero” limitation covers a mathematical concept of modifying an absolute value of an activation in the tensor. Such modification is discussed in paragraph 216 of the specification, further providing evidence that the claimed pruning the tensor by modifying an absolute value is itself a mathematical concept.
The “determining an accuracy of the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine an observed accuracy of the generically-recited neural network (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining a second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “a second activation threshold based on the first activation threshold and the accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “modifying the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) based on observed data in the dataset.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The claim recites:
“One or more non-transitory computer-readable media storing instructions executable to perform operations for modifying a neural network, the operations comprising:
inputting a dataset into the neural network, the neural network comprising a plurality of layers;”
The judicial exceptions are not integrated into a practical application. In particular, the claim recites the “One or more non-transitory computer-readable media” limitation which amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer or merely uses a computer as a tool to perform an abstract idea (i.e., generic computer components – a generic computer-readable media, performing generic computer functions which does not integrate a judicial exception into a practical application. See MPEP 2106.05(f).
The “inputting a dataset into the neural network” limitation describes data gathering with a dataset as the data. This inputting a dataset limitation can be characterized as insignificant extra-solution activity (i.e., data gathering). See MPEP 2106.05(g).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The “One or more non-transitory computer-readable media” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element represents mere instructions to apply an exception (i.e., the additional element recites one or more non-transitory computer-readable media storing instructions for applying the abstract ideas). Mere instructions to store instructions do not amount to significantly more than the judicial exception. As noted above, merely asserting that a judicial exception is to be carried out on a generic non-transitory computer-readable media cannot provide significantly more than the judicial exception. Mere instructions to apply an exception cannot provide an inventive concept.
The “inputting a dataset into the neural network” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, recitations of “inputting a dataset into the neural network” are well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
This claim is not patent eligible.
Regarding claim 12, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 12 is directed to one or more computer-readable media as depending from claim 11, thus the analysis for patent eligibility of claim 11 is incorporated herein.
Step 2A Prong 1: The claim recites:
“determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network;
determine whether the accuracy loss exceeds a threshold; and
in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold.”
The “determining the second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “an accuracy loss caused by pruning the tensor based on the accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determine … the accuracy loss” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “whether the accuracy loss exceeds a threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining that the accuracy loss is lower than the threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold by increasing the first activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 13, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 13 is directed to one or more computer-readable media as depending from claim 12, thus the analysis for patent eligibilities of claim 12 and of base claim 11 are incorporated herein.
Step 2A Prong 1: The claim recites:
“in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold.”
The “determining that the accuracy loss exceeds the threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold by decreasing the first activation threshold”(corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 14, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 14 is directed to one or more computer-readable media as depending from claim 11, thus the analysis for patent eligibilities/eligibility of claim 11 and is incorporated herein.
Step 2A Prong 1: The claim recites:
“determining the first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.”
The “determining the first activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the first activation threshold based on a third activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 15, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 15 is directed to one or more computer-readable media as depending from claim 14, thus the analysis for patent eligibilities of claim 14 and of base claim 11 are incorporated herein.
Step 2A Prong 1: The claim recites:
“pruning the additional tensor based on the third activation threshold;
determining an additional accuracy of the neural network based on an additional output generated by the neural network based on the pruned additional tensor; and
determining the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.”
The “pruning the additional tensor” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “which tensor to prune.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining an additional accuracy of the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “an additional accuracy of the neural network based on an additional output generated by the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining the first activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the first activation threshold based on the third activation threshold and the additional accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The claim recites:
“inputting an additional dataset into the neural network, the selected layer in the neural network generating an additional tensor based on the additional dataset.”
The “inputting an additional dataset into the neural network” limitation describes data gathering with “an additional dataset” as the data. This inputting an additional dataset into the neural network limitation can be characterized as insignificant extra-solution activity (i.e., data gathering). See MPEP 2106.05(g).
Accordingly, these/this additional element(s) do/does not integrate the abstract idea into a practical application because <they do not/ it does not> impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The “inputting an additional dataset into the neural network” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, recitations of “inputting an additional dataset into the neural network, the selected layer in the neural network generating an additional tensor” are well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
This claim is not patent eligible.
Regarding claim 16, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 16 is directed to one or more computer-readable media as depending from claim 11, thus the analysis for patent eligibility of claim 11 is incorporated herein.
Step 2A Prong 1: The claim recites:
“selecting another layer in the neural network; and
modifying the neural network by adding another activation pruning operation to the another layer, the another activation pruning operation to prune one or more tensors to be generated by the another layer based on another activation threshold.”
The “selecting another layer in the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine the layer to prune in the neural network (corresponding to mental processes which can be done mentally or by pen and paper).
The “modifying the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “based on another activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 17, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 17 is directed to one or more computer-readable media as depending from claim 11, thus the analysis for patent eligibility of claim 11 is incorporated herein.
Step 2A Prong 1: The claim recites:
“selecting the layer based on an amount of internal parameters of the layer, an amount of computations in the layer, a type of the layer, or some combination thereof.”
The “selecting the layer” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the layer based on an amount of internal parameters of the layer.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 18, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 18 is directed to a one or more computer-readable media as depending from claim 11, thus the analysis for patent eligibility of claim 11 is incorporated herein.
Step 2A Prong 1: The claim recites:
“further modifying the neural network by adding a weight pruning operation to the selected layer, the activation pruning operation to prune a kernel of the selected the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero,
wherein the absolute value of the weight is lower than the weight threshold.”
The “modifying the neural network” limitation added by this claim covers a mathematical concept of modifying an absolute value of a weight in the kernel to zero. Such modification is discussed of this step in paragraph 223 of the specification, further providing evidence that the activation pruning operation to prune a kernel of the selected the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero is itself a mathematical concept.
The “absolute value of the weight” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “if the weight is lower than the weight threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 19, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 19 is directed to one or more computer-readable media as depending from claim 18, thus the analysis for patent eligibility of claim 18 is incorporated herein.
Step 2A Prong 1: The claim recites:
“to determine the absolute value of the weight”
The “determine the absolute value” limitation added by this claim covers a mathematical concept of determining the absolute value of the weight. Such determination is discussed in paragraph 224 of the specification, further providing evidence that the claimed neural network training involving determining the absolute value of the weight is itself a mathematical concept.
Step 2A Prong 2: The claim recites:
“wherein the neural network has been trained”
The judicial exceptions are not integrated into a practical application. In particular, the claim recites the “neural network has been trained” limitation which amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer or merely uses a computer as a tool to perform an abstract idea (i.e., generic computer components – generic neural network performing generic computer functions) which does not integrate a judicial exception into a practical application. See MPEP 2106.05(f).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The “neural network has been trained” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element represents mere instructions to apply an exception (i.e., the additional element recites “wherein the neural network has been trained” for applying the abstract ideas). Mere instructions to train the neural network do not amount to significantly more than the judicial exception. As noted above, merely asserting that a judicial exception is to be carried out on a generic neural network cannot provide significantly more than the judicial exception. Mere instructions to apply an exception cannot provide an inventive concept.
Regarding claim 20, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 20 is directed to one or more computer-readable media as depending from claim 18, thus the analysis for patent eligibility of claim 18 is incorporated herein.
Step 2A Prong 1: The claim recites:
“determining the second activation threshold after adding the weight pruning operation to the layer.”
The “determining the second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding independent claim 21, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 21 is directed to an apparatus, corresponding to an article of manufacture, which is one of the statutory categories.
Step 2A Prong 1: The claim recites
“selecting a layer from the plurality of layers in the neural network, the layer in the neural network generating a tensor based on the dataset;
pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold;
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor;
determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold;
modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold.”
The “selecting a layer from the plurality of layers” limitation, as drafted, under its BRI, covers concepts performed in the human mind including an observation, evaluation, judgement, or opinion to select a layer in the generically-recited neural network in order to generate/populate a tensor based on observed data in “the dataset.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “pruning the tensor … by modifying an absolute value of an activation in the tensor to zero” limitation covers a mathematical concept of modifying an absolute value of an activation in the tensor. Such modification is discussed in paragraph 216 of the specification, further providing evidence that the claimed pruning the tensor by modifying an absolute value is itself a mathematical concept.
The “determining an accuracy of the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine an observed accuracy of the generically-recited neural network (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining a second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “a second activation threshold based on the first activation threshold and the accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “modifying the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) based on observed data in the dataset.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The claim recites:
“a computer processor for executing computer program instructions; and
a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations for modifying a neural network, the operations comprising:
inputting a dataset into the neural network, the neural network comprising a plurality of layers;”
The judicial exceptions are not integrated into a practical application. In particular, the claim recites the “computer processor” and “non-transitory computer-readable memory” limitations which amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer or merely uses a computer as a tool to perform an abstract idea (i.e., generic computer components – a generic processor and memory, performing generic computer functions which does not integrate a judicial exception into a practical application. See MPEP 2106.05(f).
The “inputting a dataset into the neural network” limitation describes data gathering with a dataset as the data. This inputting a dataset limitation can be characterized as insignificant extra-solution activity (i.e., data gathering). See MPEP 2106.05(g).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The “computer processor” and “non-transitory computer-readable memory” limitations do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element represents mere instructions to apply an exception (i.e., the additional elements recite a computer processor for executing computer program instructions and a non-transitory computer-readable memory storing computer program instructions for applying the abstract ideas). Mere instructions to execute and store do not amount to significantly more than the judicial exception. As noted above, merely asserting that a judicial exception is to be carried out on a generic processor and memory cannot provide significantly more than the judicial exception. Mere instructions to apply an exception cannot provide an inventive concept.
The “inputting a dataset into the neural network” limitation does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, recitations of “inputting a dataset into the neural network” are well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
This claim is not patent eligible.
Regarding claim 22, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 22 is directed to a machine and manufacture as depending from claim 11, thus the analysis for patent eligibility of claim 11 is incorporated herein.
Step 2A Prong 1: The claim recites:
“determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network;
determine whether the accuracy loss exceeds a threshold;
in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold; and
“in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold.”
The “determining the second activation threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “an accuracy loss caused by pruning the tensor based on the accuracy of the neural network.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determine … the accuracy loss” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “whether the accuracy loss exceeds a threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining that the accuracy loss is lower than the threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold by increasing the first activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
The “determining that the accuracy loss exceeds the threshold” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “the second activation threshold by decreasing the first activation threshold”(corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible
Regarding claim 24, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 24 is directed to a machine and manufacture as depending from claim 21, thus the analysis for patent eligibility of claim 21 is incorporated herein.
Step 2A Prong 1: The claim recites:
“further modifying the neural network by adding a weight pruning operation to the layer, the activation pruning operation to prune a kernel the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero,
wherein the absolute value of the weight is lower than the weight threshold.”
The “modifying the neural network” limitation added by this claim covers a mathematical concept of modifying an absolute value of a weight in the kernel to zero. Such modification is discussed in paragraph 223 of the specification, further providing evidence that the claimed pruning operation by modifying an absolute value of a weight in the kernel to zero which is itself a mathematical concept.
The “absolute value of the weight” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “if the weight is lower than the weight threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Regarding claim 25, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 25 is directed to a machine and manufacture as depending from claim 21, thus the analysis for patent eligibility of claim 21 is incorporated herein.
Step 2A Prong 1: The claim recites:
“selecting another layer in the neural network; and
modifying the neural network by adding another activation pruning operation to the another layer, the another activation pruning operation to prune one or more tensors to be generated by the another layer based on another activation threshold.”
The “selecting another layer in the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine the layer to prune in the neural network (corresponding to mental processes which can be done mentally or by pen and paper).
The “modifying the neural network” limitation, as drafted, under its BRI, covers concepts performed in the human mind (including an observation, evaluation, judgement, or opinion) to determine “based on another activation threshold.” (corresponding to mental processes which can be done mentally or by pen and paper).
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B: The claim does not recite additional elements that are sufficient to amount to significantly more than the judicial exception.
This claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 11, and 21, are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. U.S. Publication No. 20210117781, hereinafter “Kim” in view of Dally et al. U.S. Publication No. 20180046916, hereinafter “Dally” and further in view of De’souza et al. U.S. Publication No. 20230252165, hereinafter “De’souza” and even further in view of Weiss et al. U.S. Publication No. 20160350655, hereinafter “Weiss”.
Regarding claim 1, Kim discloses the method as claimed including inputting a dataset into the neural network, the neural network comprising a plurality of layers (see, paragraph 56, “The neural network 1 implemented by a DNN architecture may include a plurality of layers that process effective information. Thus, the neural network 1 may process more complex data sets than neural networks including a single layer.” [i.e., inputting a dataset into a neural network/neural network may process more complex datasets with a neural network including layers]);
selecting a layer from the plurality of layers in the neural network, the selected layer in the neural network generating a tensor based on the dataset (see, paragraph 68, “the like using any one or any combination of the neural network layers and/or neural networks made up of one or more of the layers of nodal convolutional interactions discussed herein, such as smart phones, tablet devices, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous driving vehicles, robot devices, and medical devices, without being limited thereto. Furthermore, the neural network device 300 may be a hardware (HW) accelerator dedicated for implementing or driving the above-described devices or a hardware accelerator dedicated for implementing or driving a neural network, such as a neural processing unit (NPU), a tensor processing unit (TPU), and a neural engine, without being limited thereto.” [i.e., selecting a layer from the layers in the neural network/using any one or any combination of the neural network layers, the selected layer in the neural network dedicated for implementing or driving neural network such as a tensor processing unit (TPU)]);
Although Kim substantially discloses the claimed invention, Kim does not explicitly disclose pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold;
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor.
In the same field, analogous art Dally teaches pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold (see, paragraphs 46 and 47, “The primary technique for creating weight sparsity is to prune the network during training. In one embodiment, any weight with an absolute value that is close to zero (e.g. below a defined threshold) is set to zero. The pruning process has the effect of removing weights from the filters, and sometimes even forcing an output activation to always equal zero” and “input activations having an absolute value below a defined threshold are set to zero.” [i.e., pruning the tensor/prune the network and a first activation threshold by modifying an absolute value of an activation in the tensor to zero/ absolute value that is close to zero (e.g. below a defined threshold) is set to zero wherein the absolute value of the activation is lower than the first activation threshold/ input activations having an absolute value below a defined threshold are set to zero]);
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor (see, paragraph 46, “Sparsity in a layer of a CNN is defined as the fraction of zeros in the layer's weight and input activation matrices. The primary technique for creating weight sparsity is to prune the network during training. In one embodiment, any weight with an absolute value that is close to zero (e.g. below a defined threshold) is set to zero. The pruning process has the effect of removing weights from the filters, and sometimes even forcing an output activation to always equal zero. The remaining network may be retrained, to regain the accuracy lost through naïve pruning. The result is a smaller network with accuracy extremely close to the original network. The process can be iteratively repeated to reduce network size while maintaining accuracy.” [i.e., determining an accuracy of the neural network/determining an “accuracy extremely close to the original network” based on the pruned tensor/through naïve pruning]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Dally so that the pruning process has the effect of removing weights from the filters and forcing output activation to equal zero (see Dally, e.g., paragraph 46). Doing so would have allowed Kim to use Dally ‘s result of a smaller network with accuracy extremely close to the original network. The process can be iteratively repeated to reduce network size while maintaining accuracy, as suggested by Dally (see Dally, paragraph 46). It would have also been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Dally so that the remaining network can be retrained, to regain the accuracy lost through naïve pruning. The result is a smaller network with accuracy extremely close to the original network (see Dally, e.g., paragraph 46). Doing so would have allowed Kim to use Dally‘s, pruning for the reduction of network size while maintaining accuracy which is efficient and effective for the operation of the neural network as suggested by Dally, (see Dally, paragraph 46).
Although Kim in view of Dally substantially teach the claimed invention, Kim in view of Dally do not explicitly teach determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold.
In the same field, analogous art De’souza teaches determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold (see, paragraphs 561 and 174, “a first threshold amount may be used for a first source and second threshold amount different than the first threshold amount may be used for a second source” and “evaluating including using a holdback portion of the data not used in training in order to evaluate model accuracy on the holdout data. This lifecycle may apply for any type of parallelized machine learning, not just neural networks or deep learning” [i.e., determining a second activation threshold … having a different value from the first activation threshold/ second threshold amount different than the first threshold amount and accuracy of the neural network/model accuracy]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally to incorporate the teachings of De’souza so that the second threshold amount is different than the first threshold amount and may be used for a second source (see De’souza, e.g., paragraph 561). Doing so would have allowed Kim in view of Dally to use De’souza‘s source to typically provide write requests for data that is already encrypted, the threshold amount may be set to be relatively low or disabled entirely for data protection system 400 which may set the threshold amount based on one or more attributes of a source of the data, as suggested by De’souza (see De’souza, paragraph 561).
Although Kim in view of Dally and De’souza substantially teach the claimed invention, Kim in view of Dally and De’souza do not explicitly teach modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold.
In the same field, analogous art Weiss teaches modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold (see, paragraph 98, “shrinking the network (also known as ‘Pruning’) may remove artificial ‘neurons’/cells of the neural network, upon their value dropping beyond a certain threshold” [i.e., Modifying the neural network by pruning/shrinking/pruning the network and based on activation threshold/upon value dropping beyond a certain threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Weiss so that shrinking the network (also known as ‘Pruning’) may remove artificial ‘neurons’/cells of the neural network (see Weiss, e.g., paragraph 98). Doing so would have allowed Kim to use Weiss’, reduction of the number of ‘neurons’/cells in the neural network that need to be trained allowing for a faster and more efficient learning process, as suggested by Weiss (see Weiss, paragraph 98).
Regarding claim 11, Kim discloses the one or more non-transitory computer-readable media as claimed including inputting a dataset into the neural network, the neural network comprising a plurality of layers (see, paragraph 56, “The neural network 1 implemented by a DNN architecture may include a plurality of layers that process effective information. Thus, the neural network 1 may process more complex data sets than neural networks including a single layer.” [i.e., inputting a dataset into a neural network/neural network may process more complex datasets with neural network comprising a plurality of layers/neural network … may include a plurality of layers]);
selecting a layer from the plurality of layers in the neural network, the selected layer in the neural network generating a tensor based on the dataset (see, paragraph 68, “the like using any one or any combination of the neural network layers and/or neural networks made up of one or more of the layers of nodal convolutional interactions discussed herein, such as smart phones, tablet devices, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous driving vehicles, robot devices, and medical devices, without being limited thereto. Furthermore, the neural network device 300 may be a hardware (HW) accelerator dedicated for implementing or driving the above-described devices or a hardware accelerator dedicated for implementing or driving a neural network, such as a neural processing unit (NPU), a tensor processing unit (TPU), and a neural engine, without being limited thereto.” [i.e., selecting a layer from the plurality of layers in the neural network/using any one or any combination of the neural network layers and the selected layer in the neural network generating a tensor based on the dataset/the tensor(neural network device)dedicated for implementing or driving neural network such as … a tensor processing unit (TPU)]);
Although Kim substantially discloses the claimed invention, Kim does not explicitly disclose pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold;
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor.
In the same field, analogous art Dally teaches pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold (see, paragraphs 46 and 47, “The primary technique for creating weight sparsity is to prune the network during training. In one embodiment, any weight with an absolute value that is close to zero (e.g. below a defined threshold) is set to zero. The pruning process has the effect of removing weights from the filters, and sometimes even forcing an output activation to always equal zero” and “input activations having an absolute value below a defined threshold are set to zero.” [i.e., pruning the tensor/prune the network and a first activation threshold by modifying an absolute value of an activation in the tensor to zero/ absolute value that is close to zero (e.g. below a defined threshold) is set to zero wherein the absolute value of the activation is lower than the first activation threshold/ input activations having an absolute value below a defined threshold are set to zero]);
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor (see, paragraph 46, “Sparsity in a layer of a CNN is defined as the fraction of zeros in the layer's weight and input activation matrices. The primary technique for creating weight sparsity is to prune the network during training. In one embodiment, any weight with an absolute value that is close to zero (e.g. below a defined threshold) is set to zero. The pruning process has the effect of removing weights from the filters, and sometimes even forcing an output activation to always equal zero. The remaining network may be retrained, to regain the accuracy lost through naïve pruning. The result is a smaller network with accuracy extremely close to the original network. The process can be iteratively repeated to reduce network size while maintaining accuracy.” [i.e., determining an accuracy of the neural network/determining an “accuracy extremely close to the original network” based on the pruned tensor/through naïve pruning]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Dally so that the pruning process has the effect of removing weights from the filters and forcing output activation to equal zero (see Dally, e.g., paragraph 46). Doing so would have allowed Kim to use Dally ‘s result of a smaller network with accuracy extremely close to the original network. The process can be iteratively repeated to reduce network size while maintaining accuracy, as suggested by Dally (see Dally, paragraph 46). It would have also been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Dally so that the remaining network to be retrained, to regain the accuracy lost through naïve pruning. The result is a smaller network with accuracy extremely close to the original network (see Dally, e.g., paragraph 46). Doing so would have allowed Kim to use Dally‘s, process for iterative repetition to reduce network size while maintaining accuracy as suggested by Dally, (see Dally, paragraph 46).
Although Kim in view of Dally substantially teach the claimed invention, Kim in view of Dally do not explicitly teach determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold.
In the same field, analogous art De’souza teaches determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold (see, paragraphs 561 and 174, “a first threshold amount may be used for a first source and second threshold amount different than the first threshold amount may be used for a second source” and “evaluating including using a holdback portion of the data not used in training in order to evaluate model accuracy on the holdout data. This lifecycle may apply for any type of parallelized machine learning, not just neural networks or deep learning” [i.e., determining a second activation threshold … having a different value from the first activation threshold/ second threshold amount different than the first threshold amount and accuracy of the neural network/model accuracy]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally to incorporate the teachings of De’souza so that the second threshold amount is different than the first threshold amount and may be used for a second source (see De’souza, e.g., paragraph 561). Doing so would have allowed Kim in view of Dally to use De’souza‘s source to typically provide write requests for data that is already encrypted, the threshold amount may be set to be relatively low or disabled entirely for data protection system 400 which may set the threshold amount based on one or more attributes of a source of the data, as suggested by De’souza (see De’souza, paragraph 561).
Although Kim in view of Dally and De’souza substantially teach the claimed invention, Kim in view of Dally and De’souza do not explicitly teach modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold.
In the same field, analogous art Weiss teaches modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold (see, paragraph 98, “shrinking the network (also known as ‘Pruning’) may remove artificial ‘neurons’/cells of the neural network, upon their value dropping beyond a certain threshold” [i.e., Modifying the neural network by pruning/(shrinking/pruning) the network and based on activation threshold/upon value dropping beyond a certain threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Weiss so that shrinking the network (also known as ‘Pruning’) may remove artificial ‘neurons’/cells of the neural network (see Weiss, e.g., paragraph 98). Doing so would have allowed Kim to use Weiss’, shrinking the network to lessen the number of ‘neurons’/cells that need to be trained allowing for a faster learning process, as suggested by Weiss (see Weiss, paragraph 98).
Regarding claim 21, Kim discloses an apparatus as claimed including inputting a dataset into the neural network, the neural network comprising a plurality of layers (see, paragraph 56, “The neural network 1 implemented by a DNN architecture may include a plurality of layers that process effective information. Thus, the neural network 1 may process more complex data sets than neural networks including a single layer.” [i.e., inputting a dataset into a neural network/neural network may process more complex datasets with neural network comprising a plurality of layers/neural network … may include a plurality of layers]);
selecting a layer from the plurality of layers in the neural network, the selected layer in the neural network generating a tensor based on the dataset (see, paragraph 68, “the like using any one or any combination of the neural network layers and/or neural networks made up of one or more of the layers of nodal convolutional interactions discussed herein, such as smart phones, tablet devices, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous driving vehicles, robot devices, and medical devices, without being limited thereto. Furthermore, the neural network device 300 may be a hardware (HW) accelerator dedicated for implementing or driving the above-described devices or a hardware accelerator dedicated for implementing or driving a neural network, such as a neural processing unit (NPU), a tensor processing unit (TPU), and a neural engine, without being limited thereto.” [i.e., selecting a layer from the plurality of layers in the neural network/using any one or any combination of the neural network layers and the selected layer in the neural network generating a tensor based on the dataset/the tensor(neural network device)dedicated for implementing or driving neural network such as … a tensor processing unit (TPU)]);
Although Kim substantially discloses the claimed invention, Kim does not explicitly disclose pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold;
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor.
In the same field, analogous art Dally teaches pruning the tensor based on a first activation threshold by modifying an absolute value of an activation in the tensor to zero, wherein the absolute value of the activation is lower than the first activation threshold (see, paragraphs 46 and 47, “The primary technique for creating weight sparsity is to prune the network during training. In one embodiment, any weight with an absolute value that is close to zero (e.g. below a defined threshold) is set to zero. The pruning process has the effect of removing weights from the filters, and sometimes even forcing an output activation to always equal zero” and “input activations having an absolute value below a defined threshold are set to zero.” [i.e., pruning the tensor/prune the network and a first activation threshold by modifying an absolute value of an activation in the tensor to zero/ absolute value that is close to zero (e.g. below a defined threshold) is set to zero wherein the absolute value of the activation is lower than the first activation threshold/ input activations having an absolute value below a defined threshold are set to zero]);
determining an accuracy of the neural network based on an output of the neural network, the neural network generating the output based on the pruned tensor (see, paragraph 46, “Sparsity in a layer of a CNN is defined as the fraction of zeros in the layer's weight and input activation matrices. The primary technique for creating weight sparsity is to prune the network during training. In one embodiment, any weight with an absolute value that is close to zero (e.g. below a defined threshold) is set to zero. The pruning process has the effect of removing weights from the filters, and sometimes even forcing an output activation to always equal zero. The remaining network may be retrained, to regain the accuracy lost through naïve pruning. The result is a smaller network with accuracy extremely close to the original network. The process can be iteratively repeated to reduce network size while maintaining accuracy.” [i.e., determining an accuracy of the neural network/determining an “accuracy extremely close to the original network” based on the pruned tensor/through naïve pruning]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Dally so that the pruning process has the effect of removing weights from the filters and forcing output activation to equal zero (see Dally, e.g., paragraph 46). Doing so would have allowed Kim to use Dally ‘s result of a smaller network with accuracy extremely close to the original network. The process can be iteratively repeated to reduce network size while maintaining accuracy, as suggested by Dally (see Dally, paragraph 46). It would have also been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Dally so that the remaining network to be retrained, to regain the accuracy lost through naïve pruning. The result is a smaller network with accuracy extremely close to the original network (see Dally, e.g., paragraph 46). Doing so would have allowed Kim to use Dally‘s, process for iterative repetition to reduce network size while maintaining accuracy as suggested by Dally, (see Dally, paragraph 46).
Although Kim in view of Dally substantially teach the claimed invention, Kim in view of Dally do not explicitly teach determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold.
In the same field, analogous art De’souza teaches determining a second activation threshold based on the first activation threshold and the accuracy of the neural network, the second activation threshold having a different value from the first activation threshold (see, paragraphs 561 and 174, “a first threshold amount may be used for a first source and second threshold amount different than the first threshold amount may be used for a second source” and “evaluating including using a holdback portion of the data not used in training in order to evaluate model accuracy on the holdout data. This lifecycle may apply for any type of parallelized machine learning, not just neural networks or deep learning” [i.e., determining a second activation threshold … having a different value from the first activation threshold/ second threshold amount different than the first threshold amount and accuracy of the neural network/model accuracy]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally to incorporate the teachings of De’souza so that the second threshold amount is different than the first threshold amount and may be used for a second source (see De’souza, e.g., paragraph 561). Doing so would have allowed Kim in view of Dally to use De’souza‘s source to typically provide write requests for data that is already encrypted, the threshold amount may be set to be relatively low or disabled entirely for data protection system 400 which may set the threshold amount based on one or more attributes of a source of the data, as suggested by De’souza (see De’souza, paragraph 561).
Although Kim in view of Dally and De’souza substantially teach the claimed invention, Kim in view of Dally and De’souza do not explicitly teach modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold.
In the same field, analogous art Weiss teaches modifying the neural network by adding an activation pruning operation to the layer, the activation pruning operation to prune one or more tensors to be generated by the layer based on the second activation threshold (see, paragraph 98, “shrinking the network (also known as ‘Pruning’) may remove artificial ‘neurons’/cells of the neural network, upon their value dropping beyond a certain threshold” [i.e., Modifying the neural network by pruning/(shrinking/pruning) the network and based on activation threshold/upon value dropping beyond a certain threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim to incorporate the teachings of Weiss so that shrinking the network (also known as ‘Pruning’) may remove artificial ‘neurons’/cells of the neural network (see Weiss, e.g., paragraph 98). Doing so would have allowed Kim to use Weiss’, shrinking the network to lessen the number of ‘neurons’/cells that need to be trained allowing for a faster learning process, as suggested by Weiss (see Weiss, paragraph 98).
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Dally, De’souza, and Weiss as applied to claims 1 and 11 above and further in view of non-patent literature Hoefler et al. (“Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks”, hereinafter “Hoefler”) and additionally in view of Li et al. (U.S. Publication No. 20190057477, hereinafter “Li”).
Regarding claim 2, as discussed above, Kim in view of Dally, De’souza, and Weiss teach the method of claim 1.
Although Kim in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim in view of Dally, De’souza, and Weiss do not explicitly teach determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network;
determine whether the accuracy loss exceeds a threshold.
In the same field, analogous art Hoefler teaches determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network (see, page 56, section 6.2.1, “while attention heads are important for training, several works showed that most of the heads can be pruned after training with only minor accuracy loss … Michel et al. [2019] show similar results with a first-order head importance score for pruning. Using an iterative greedy process to test model quality with each head removed, they are able to prune 20–40% of attention heads with an insignificant decrease in quality. They also find that the importance of heads is transferable across tasks and that the importance” [i.e., accuracy caused by pruning the tensor/(attention) heads can be pruned after training … with only minor accuracy loss based on accuracy of the neural network/first-order head importance score for pruning, using a greedy process to test model (neural network) quality]);
determine whether the accuracy loss exceeds a threshold (see, page 70, fig. 26, “Relative validation ImageNet accuracy loss for different pruning densities, strategies, and neural networks. Solid lines represent best-performing networks, whereas dotted lines represent accuracy thresholds (e.g., 1% relative accuracy reduction is the maximum allowed by MLPerf ImageNet rules [Mattson et al. 2020]).” [i.e., Accuracy loss as it grows larger from left to right indicates the loss becoming greater and if further right past the threshold, it indicates that the accuracy loss exceeds the threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally, De’souza, and Weiss to incorporate the teachings of Hoefler so that while attention heads are important for training, several works showed that most of the heads can be pruned after training with only minor accuracy loss (see Hoefler, page 56, section 6.2.1). Doing so would have allowed Kim in view of Dally, De’souza, and Weiss to use Hoefler‘s recognition that growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components, as suggested by Hoefler (see Hoefler, page 56, section 6.2.1). It would also have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally, De’souza, and Weiss to incorporate the teachings of Hoefler so that solid lines represent best-performing networks, whereas dotted lines represent accuracy thresholds to see if accuracy loss becomes greater as it moves right past the threshold (see Hoefler, e.g., page 70, fig. 26). Doing so would have allowed Kim in view of Dally, De’souza, and Weiss to use Hoefler‘s negative accuracy drop means improvement in generalization, as suggested by Hoefler (see Hoefler, page 70, fig. 26).
Although Kim in view of Dally, De’souza, Weiss and Hoefler substantially teach the claimed invention, Kim in view of Dally, De’souza, Weiss and Hoefler do not explicitly teach in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold.
In the same field, analogous art Li teaches in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold (see, paragraph 26, “activation unit 110 can increase the activation threshold from a first activation threshold of queue 124 to a second activation threshold of queue 124′. It is contemplated that, the second activation threshold can be determined based on the first activation threshold” [i.e., determining the second activation threshold by increasing the first activation threshold/increase the activation threshold from the first activation threshold … the second activation threshold can be determined based on the first activation threshold]).
Regarding claim 12, as discussed above, Kim in view of Dally, De’souza, and Weiss teach the one or more non-transitory computer-readable media of claim 11.
Although Kim in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim in view of Dally, De’souza, and Weiss do not explicitly teach determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network;
determine whether the accuracy loss exceeds a threshold.
In the same field, analogous art Hoefler teaches determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network (see, page 56, section 6.2.1, “while attention heads are important for training, several works showed that most of the heads can be pruned after training with only minor accuracy loss … Michel et al. [2019] show similar results with a first-order head importance score for pruning. Using an iterative greedy process to test model quality with each head removed, they are able to prune 20–40% of attention heads with an insignificant decrease in quality. They also find that the importance of heads is transferable across tasks and that the importance” [i.e., accuracy caused by pruning the tensor/(attention) heads can be pruned after training … with only minor accuracy loss based on accuracy of the neural network/first-order head importance score for pruning, using a greedy process to test model (neural network) quality]);
determine whether the accuracy loss exceeds a threshold (see, page 70, fig. 26, “Relative validation ImageNet accuracy loss for different pruning densities, strategies, and neural networks. Solid lines represent best-performing networks, whereas dotted lines represent accuracy thresholds (e.g., 1% relative accuracy reduction is the maximum allowed by MLPerf ImageNet rules [Mattson et al. 2020]).” [i.e., Accuracy loss as it grows larger from left to right indicates the loss becoming greater and if further right past the threshold, it indicates that the accuracy loss exceeds the threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally, De’souza, and Weiss to incorporate the teachings of Hoefler so that while attention heads are important for training, several works showed that most of the heads can be pruned after training with only minor accuracy loss (see Hoefler, page 56, section 6.2.1). Doing so would have allowed Kim in view of Dally, De’souza, and Weiss to use Hoefler‘s recognition that growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components, as suggested by Hoefler (see Hoefler, page 56, section 6.2.1). It would also have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally, De’souza, and Weiss to incorporate the teachings of Hoefler so that solid lines represent best-performing networks, whereas dotted lines represent accuracy thresholds to see if accuracy loss becomes greater as it moves right past the threshold (see Hoefler, e.g., page 70, fig. 26). Doing so would have allowed Kim in view of Dally, De’souza, and Weiss to use Hoefler‘s negative accuracy drop means improvement in generalization, as suggested by Hoefler (see Hoefler, page 70, fig. 26).
Although Kim in view of Dally, De’souza, Weiss and Hoefler substantially teach the claimed invention, Kim in view of Dally, De’souza, Weiss and Hoefler do not explicitly teach in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold.
In the same field, analogous art Li teaches in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold (see, paragraph 26, “activation unit 110 can increase the activation threshold from a first activation threshold of queue 124 to a second activation threshold of queue 124′. It is contemplated that, the second activation threshold can be determined based on the first activation threshold” [i.e., determining the second activation threshold by increasing the first activation threshold/increase the activation threshold from the first activation threshold … the second activation threshold can be determined based on the first activation threshold]).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Dally, De’souza, and Weiss as applied to claim 21 above and further in view of non-patent literature Hoefler and additionally in view of Li at el.
Although Kim in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim in view of Dally, De’souza, and Weiss do not explicitly teach determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network;
determine whether the accuracy loss exceeds a threshold.
In the same field, analogous art Hoefler teaches determining an accuracy loss caused by pruning the tensor based on the accuracy of the neural network (see, page 56, section 6.2.1, “while attention heads are important for training, several works showed that most of the heads can be pruned after training with only minor accuracy loss … Michel et al. [2019] show similar results with a first-order head importance score for pruning. Using an iterative greedy process to test model quality with each head removed, they are able to prune 20–40% of attention heads with an insignificant decrease in quality. They also find that the importance of heads is transferable across tasks and that the importance” [i.e., accuracy caused by pruning the tensor/(attention) heads can be pruned after training … with only minor accuracy loss based on accuracy of the neural network/first-order head importance score for pruning, using a greedy process to test model (neural network) quality]);
determine whether the accuracy loss exceeds a threshold (see, page 70, fig. 26, “Relative validation ImageNet accuracy loss for different pruning densities, strategies, and neural networks. Solid lines represent best-performing networks, whereas dotted lines represent accuracy thresholds (e.g., 1% relative accuracy reduction is the maximum allowed by MLPerf ImageNet rules [Mattson et al. 2020]).” [i.e., Accuracy loss as it grows larger from left to right indicates the loss becoming greater and if further right past the threshold, it indicates that the accuracy loss exceeds the threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally, De’souza, and Weiss to incorporate the teachings of Hoefler so that while attention heads are important for training, several works showed that most of the heads can be pruned after training with only minor accuracy loss (see Hoefler, page 56, section 6.2.1). Doing so would have allowed Kim in view of Dally, De’souza, and Weiss to use Hoefler‘s recognition that growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components, as suggested by Hoefler (see Hoefler, page 56, section 6.2.1). It would also have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally, De’souza, and Weiss to incorporate the teachings of Hoefler so that solid lines represent best-performing networks, whereas dotted lines represent accuracy thresholds to see if accuracy loss becomes greater as it moves right past the threshold (see Hoefler, e.g., page 70, fig. 26). Doing so would have allowed Kim in view of Dally, De’souza, and Weiss to use Hoefler‘s negative accuracy drop means improvement in generalization, as suggested by Hoefler (see Hoefler, page 70, fig. 26).
Although Kim in view of Dally, De’souza, Weiss and Hoefler substantially teach the claimed invention, Kim in view of Dally, De’souza, Weiss and Hoefler do not explicitly teach in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold.
in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold.
In the same field, analogous art Li teaches in response to determining that the accuracy loss is lower than the threshold, determining the second activation threshold by increasing the first activation threshold (see, paragraph 26, “activation unit 110 can increase the activation threshold from a first activation threshold of queue 124 to a second activation threshold of queue 124′. It is contemplated that, the second activation threshold can be determined based on the first activation threshold” [i.e., determining the second activation threshold by increasing the first activation threshold/increase the activation threshold from the first activation threshold … the second activation threshold can be determined based on the first activation threshold]) and
in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold (see, paragraph 26, “It is contemplated that, the second activation threshold can be determined based on the first activation threshold, the activation number, and the number of active queues in the area” [i.e., determining the second activation threshold by decreasing the first activation threshold/the second activation threshold can be determined based on the first activation threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim in view of Dally, De’souza, Weiss and Hoefler to incorporate the teachings of Li so that the increasing or decreasing in the activation threshold from the first activation threshold … the second activation threshold can be determined based on the first activation threshold (see Li, e.g., paragraph 26). Doing so would have allowed Kim in view of Dally, De’souza, Weiss and Hoefler to use Li‘s Activation unit that can activate a response for find there in no need for a response based on exceeding or not meeting an activation threshold, as suggested by Li (see Li, paragraphs 25 and 28).
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Dally, De’souza, Weiss, Hoefler, and Li as applied to claims 2 and 12 above.
Regarding claim 3, as discussed above, Kim, in view of Dally, De’souza, Weiss, Hoefler, and Li teach the method of claim 2.
Although Kim in view of Dally, De’souza, Weiss and Hoefler substantially teach the claimed invention, Kim in view of Dally, De’souza, Weiss and Hoefler do not explicitly teach in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold.
In the same field, analogous art Li teaches in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold (see, paragraph 26, “It is contemplated that, the second activation threshold can be determined based on the first activation threshold, the activation number, and the number of active queues in the area” [i.e., determining the second activation threshold by decreasing the first activation threshold/ the second activation threshold can be determined based on the first activation threshold]).
The motivation to combine Kim, Dally, De’souza, Weiss, Hoefler and Li is the same as discussed above with respect to claim 2.
Regarding claim 13, as discussed above, Kim, in view of Dally, De’souza, Weiss, Hoefler, and Li teach the method of claim 12.
Li further teaches in response to determining that the accuracy loss exceeds the threshold, determining the second activation threshold by decreasing the first activation threshold (see, paragraph 26, “It is contemplated that, the second activation threshold can be determined based on the first activation threshold, the activation number, and the number of active queues in the area” [i.e., determining the second activation threshold by decreasing the first activation threshold/ the second activation threshold can be determined based on the first activation threshold]).
The motivation to combine Kim, Dally, De’souza, Weiss, Hoefler and Li is the same as discussed above with respect to claim 12.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, in view of Dally, De’souza, and Weiss as applied to claims 1 and 11 above and further in view of Bikumala et al. (U.S. Publication No. 20210233129, hereinafter “Bikumala”).
Regarding claim 4, as discussed above, Kim in view of Dally, De’souza, and Weiss teach the method of claim 1.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach determining the first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.
In the same field, analogous art Bikumala teaches determining the first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold (see, paragraph 20, “The operations may include determining a first set of parts comprising reliable parts having a reliability score greater than a first threshold amount. For example, the first set of parts may be used to build the product for a quality-conscious market. The operations may include determining a second set of parts comprising inexpensive parts having a reliability score greater than a second threshold amount and a price less than a third threshold amount.” [i.e., first activation threshold based on third activation threshold wherein the third activation threshold is different from the first activation threshold and the second activation threshold/first threshold amount, second as well as third threshold amount have different values otherwise price, reliability score can be compared to all three thresholds in tandem]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, Dally, De’souza, and Weiss to incorporate the teachings of Bikumala so that the first and second set of parts have a reliability score greater than a first threshold and second threshold and a price less than a third threshold (see Bikumala, e.g., paragraph 20). Doing so would have allowed Kim, Dally, De’souza, and Weiss to use Bikumala‘s different first, second and third threshold values based on scores to create a machine learning model for a specific purpose such as a machine learning taxonomy model based in part of the different scores, as suggested by Bikumala (see Bikumala, paragraph 20).
Regarding claim 14, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach one or more non-transitory computer-readable media of claim 11.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Bikumala does not explicitly teach determining the first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold.
In the same field, analogous art Bikumala teaches determining the first activation threshold based on a third activation threshold, wherein the third activation threshold is different from the first activation threshold and the second activation threshold (see, paragraph 20, “The operations may include determining a first set of parts comprising reliable parts having a reliability score greater than a first threshold amount. For example, the first set of parts may be used to build the product for a quality-conscious market. The operations may include determining a second set of parts comprising inexpensive parts having a reliability score greater than a second threshold amount and a price less than a third threshold amount.” [i.e., third activation threshold is different from the first activation threshold and the second activation threshold/first threshold amount, second as well as third threshold amount have different values otherwise price]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, Dally, De’souza, and Weiss to incorporate the teachings of Bikumala so that the first and second set of parts have a reliability score greater than a first threshold and second threshold and a price less than a third threshold (see Bikumala, e.g., paragraph 20). Doing so would have allowed Kim, Dally, De’souza, and Weiss to use Bikumala‘s different first, second and third threshold values based on scores to create a machine learning model for a specific purpose such as a machine learning taxonomy model based in part of the different scores, as suggested by Bikumala (see Bikumala, paragraph 20).
Claims 6, 16 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, Dally, De’souza, and Weiss as applied to claims 1, 11 and 21 above and further in view of Zlateski et al. (U.S. Publication No. 20200160181, hereinafter “Zlateski”).
Regarding claim 6, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach the method of claim 1.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach selecting another layer in the neural network; and
modifying the neural network by adding another activation pruning operation to the another layer, the another activation pruning operation to prune one or more tensors to be generated by the another layer based on another activation threshold.
In the same field, analogous art Zlateski teaches selecting another layer in the neural network; (see, paragraph 40, “The multiplication in the context of convolutional neural network layers is typically between kernel tensors of values, which may be sparse because as being pruned extensively, and the tensor or matrix of inputs (or modified inputs) to a NN at a first layer or inputs to an intermediate layer of a NN.” [i.e., another layer in the neural network/neural network layers]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Zlateski so that neural layers (see Zlateski, e.g., paragraph 40). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Zlateski ‘s inputs to a NN to be an image to be categorized, speech data to be analyzed, etc, as suggested by Zlateski (see Zlateski, paragraph 40).
Regarding claim 16, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach one or more non-transitory computer-readable media of claim 11.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach selecting another layer in the neural network; and
modifying the neural network by adding another activation pruning operation to the another layer, the another activation pruning operation to prune one or more tensors to be generated by the another layer based on another activation threshold.
In the same field, analogous art Zlateski teaches selecting another layer in the neural network; (see, paragraph 40, “The multiplication in the context of convolutional neural network layers is typically between kernel tensors of values, which may be sparse because as being pruned extensively, and the tensor or matrix of inputs (or modified inputs) to a NN at a first layer or inputs to an intermediate layer of a NN.” [i.e., another layer in the neural network/neural network layers]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Zlateski so that other neural layers can be selected (see Zlateski, e.g., paragraph 40). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Zlateski’s inputs to a NN for images to be a categorized and speech data to be analyzed, etc…, as suggested by Zlateski (see Zlateski, paragraph 40).
Regarding claim 25, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach an apparatus of claim 21.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach selecting another layer in the neural network; and
modifying the neural network by adding another activation pruning operation to the another layer, the another activation pruning operation to prune one or more tensors to be generated by the another layer based on another activation threshold.
In the same field, analogous art Zlateski teaches selecting another layer in the neural network (see, paragraph 40, “The multiplication in the context of convolutional neural network layers is typically between kernel tensors of values, which may be sparse because as being pruned extensively, and the tensor or matrix of inputs (or modified inputs) to a NN at a first layer or inputs to an intermediate layer of a NN.” [i.e., another layer in the neural network/neural network layers]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Zlateski so that other neural layers can be selected (see Zlateski, e.g., paragraph 40). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Zlateski ‘s inputs to a NN for images to be a categorized and speech data to be analyzed, etc …, as suggested by Zlateski (see Zlateski, paragraph 40).
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, in view of Dally, De’souza, and Weiss as applied to claims 1 and 11 above and further in view of Rezazadegan et al. (U.S. Publication No. 20220256227, hereinafter “Rezazadegan”).
Regarding claim 7, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach the method of claim 1.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach selecting the layer based on an amount of internal parameters of the layer, an amount of computations in the layer, a type of the layer, or some combination thereof.
In the same field, analogous art Rezazadegan teaches selecting the layer based on an amount of internal parameters of the layer, an amount of computations in the layer, a type of the layer, or some combination thereof (see, paragraph 482, “Topology here refers to the structure or architecture of the neural networks, such as number of layers, types of layers, number of computational units per layer, number of convolutional channels per layer, hyper-parameters of the layers, and the like.” [i.e., layers/structure or architecture of the neural networks based on parameters of the layer, computations in the layer, a type of layer/hyper-parameters of the layer, computational units per layer, types of layers]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Razazadegan so that structure or architecture of the neural networks is based on parameters of the layer, computations in the layer, a type of layers (see Razazadegan, e.g., paragraph 482). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Razazadegan‘s selection of layer for the structure of the neural network known as a topology to be used by a decoder-side device which would then proceed to use the topology and associated weights for purposes such as enhancement or as a filter of sorts, as suggested by Razazadegan (see Razazadegan, Abstract).
Regarding claim 17, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach the method of claim 11.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach selecting the layer based on an amount of internal parameters of the layer, an amount of computations in the layer, a type of the layer, or some combination thereof.
In the same field, analogous art Rezazadegan teaches selecting the layer based on an amount of internal parameters of the layer, an amount of computations in the layer, a type of the layer, or some combination thereof (see, paragraph 482, “Topology here refers to the structure or architecture of the neural networks, such as number of layers, types of layers, number of computational units per layer, number of convolutional channels per layer, hyper-parameters of the layers, and the like.” [i.e., layers/structure or architecture of the neural networks based on parameters of the layer, computations in the layer, a type of layer/hyper-parameters of the layer, computational units per layer, types of layers]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Razazadegan so that structure or architecture of the neural networks) based on (parameters of the layer, computations in the layer, a type of layers (see Razazadegan, e.g., paragraph 481). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Razazadegan‘s selection of layer for the structure of the neural network known as a topology to be used by a decoder-side device which would then proceed to use the topology and associated weights for purposes such as enhancement or as a filter of sorts, as suggested by Razazadegan (see Razazadegan, Abstract).
Claims 8, 18 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, in view of Dally, De’souza, and Weiss as applied to claims 1,11, and 21 above and further in view of Supikov et al. (U.S. Publication No. 20200117139, hereinafter “Supikov”).
Regarding claim 8, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach the method of claim 1.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach further modifying the neural network by adding a weight pruning operation to the selected layer, the activation pruning operation to prune a kernel of the selected the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero, wherein the absolute value of the weight is lower than the weight threshold.
In the same field, analogous art Supikov teaches further modifying the neural network by adding a weight pruning operation to the selected layer, the activation pruning operation to prune a kernel of the selected the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero, wherein the absolute value of the weight is lower than the weight threshold.(see, paragraphs 56 (first phrase) and 54 (latter two paragraphs/phrases), “where the pruned DNN (e.g., having fewer kernels, weights, and/or reduced accuracy weights)” and “one or more kernels of the pretrained DNN having zero or small parameters may be eliminated … Such a determination of small filter weights may be made by comparing each weight to a threshold, comparing a sum of absolute values of the weights to a threshold, comparing a sum of squares of the weights to a threshold, or requiring satisfaction of multiple of such thresholds. … use of both sum of absolute values and sum of squares may provide more robust kernel evaluation” and “convolutional kernel weights that are below a threshold may be set to zero” [i.e., modifying the neural network by weight pruning operation to the layer/pruned DNN/deep neural network having fewer weights and prune kernel based on weight and modify absolute value of weight of kernel to zero/kernel of the DNN having zero or small values/small filter weights/absolute values of the weights) may be eliminated and absolute value of weight is lower than weight threshold/kernel weights are below threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Supikov so that the pruned DNN can have fewer kernels, weights, and/or reduced accuracy weights (see Supikov e.g., paragraph 56). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Supikov ‘s compression/pruned reduction to provide an advantageously lower computational and memory footprint for the DNN along with minimal accuracy loss, as suggested by Supikov (see Supikov, paragraph 53).
Regarding claim 18, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach one or more non-transitory computer-readable media of claim 11.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach further modifying the neural network by adding a weight pruning operation to the selected layer, the activation pruning operation to prune a kernel of the selected the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero, wherein the absolute value of the weight is lower than the weight threshold.
In the same field, analogous art Supikov teaches further modifying the neural network by adding a weight pruning operation to the selected layer, the activation pruning operation to prune a kernel of the selected the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero, wherein the absolute value of the weight is lower than the weight threshold.(see, paragraph 56 (first phrase) and 54 (latter two paragraphs/phrases), “where the pruned DNN (e.g., having fewer kernels, weights, and/or reduced accuracy weights)” and “one or more kernels of the pretrained DNN having zero or small parameters may be eliminated … Such a determination of small filter weights may be made by comparing each weight to a threshold, comparing a sum of absolute values of the weights to a threshold, comparing a sum of squares of the weights to a threshold, or requiring satisfaction of multiple of such thresholds. For example, use of both sum of absolute values and sum of squares may provide more robust kernel evaluation” and “convolutional kernel weights that are below a threshold may be set to zero” [i.e., modifying the neural network by a weight pruning operation to the layer/pruned DNN (deep neural network) having fewer weights and prune kernel based on weight and modify absolute value of weight of kernel to zero/kernel of the DNN having zero or small values/small filter weights/absolute values of the weights may be eliminated and absolute value of weight is lower than weight threshold/kernel weights are below threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Supikov so that the pruned DNN can have fewer kernels, weights, and/or reduced accuracy weights (see Supikov e.g., paragraph 56). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Supikov ‘s compression/pruned reduction to provide an advantageously lower computational and memory footprint for the DNN along with minimal accuracy loss, as suggested by Supikov (see Supikov, paragraph 53).
Regarding claim 24, as discussed above, Kim, in view of Dally, De’souza, and Weiss teach an apparatus of claim 21.
Although Kim, in view of Dally, De’souza, and Weiss substantially teach the claimed invention, Kim, in view of Dally, De’souza, and Weiss do not explicitly teach further modifying the neural network by adding a weight pruning operation to the layer, the activation pruning operation to prune a kernel the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero, wherein the absolute value of the weight is lower than the weight threshold.
In the same field, analogous art Supikov teaches further modifying the neural network by adding a weight pruning operation to the layer, the activation pruning operation to prune a kernel the layer based on a weight threshold by modifying an absolute value of a weight in the kernel to zero, wherein the absolute value of the weight is lower than the weight threshold (see, paragraph 56 (first phrase) and 54 (latter two paragraphs/phrases), “where the pruned DNN (e.g., having fewer kernels, weights, and/or reduced accuracy weights)” and “one or more kernels of the pretrained DNN having zero or small parameters may be eliminated … Such a determination of small filter weights may be made by comparing each weight to a threshold, comparing a sum of absolute values of the weights to a threshold, comparing a sum of squares of the weights to a threshold, or requiring satisfaction of multiple of such thresholds. For example, use of both sum of absolute values and sum of squares may provide more robust kernel evaluation” and “convolutional kernel weights that are below a threshold may be set to zero” [i.e., modifying the neural network by a weight pruning operation to the layer/pruned DNN/deep neural network having fewer weights and prune kernel based on weight and modify absolute value of weight of kernel to zero/kernel of the DNN having zero or small values/small filter weights/absolute values of the weights may be eliminated, where the(absolute value of weight is lower than weight threshold/kernel weights are below threshold]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, and Weiss to incorporate the teachings of Supikov so that the pruned DNN can have fewer kernels, weights, and/or reduced accuracy weights (see Supikov e.g., paragraph 56). Doing so would have allowed Kim, in view of Dally, De’souza, and Weiss to use Supikov ‘s compression/pruned reduction to provide an advantageously lower computational and memory footprint for the DNN along with minimal accuracy loss, as suggested by Supikov (see Supikov, paragraph 53).
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, in view of Dally, De’souza, Weiss, and Supikov as applied to claims 8 and 18 above and further in view of Frazier et al. (U.S. Publication No. 20230169103, hereinafter “Frazier”).
Regarding claim 9, as discussed above, Kim, in view of Dally, De’souza, Weiss, and Supikov teaches the method of claim 8.
Although Kim, in view of Dally, De’souza, Weiss, and Supikov substantially teach the claimed invention, Kim, in view of Dally, De’souza, Weiss, and Supikov do not explicitly teach wherein the neural network has been trained to determine the absolute value of the weight.
In the same field, analogous art Frazier teaches wherein the neural network has been trained to determine the absolute value of the weight (see, paragraph 13, “the trained neural network comprises, at least, a dense layer optimized to predict the opportunity score to generate a sum of absolute values of edge weights” [i.e., the neural network has been trained/the trained neural network to determine the absolute value of the weight/generate a sum of absolute values of edge weights]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, Weiss, and Supikov to incorporate the teachings of Frazier so that the trained neural network to generate a sum of absolute values of edge weights (see Frazier e.g., paragraph 13). Doing so would have allowed Kim, in view of Dally, De’souza, Weiss, and Supikov to use Frazier ‘s optimized to predict the opportunity score, as suggested by Frazier (see Frazier, paragraph 13).
Regarding claim 19, as discussed above, Kim, in view of Dally, De’souza, Weiss, and Supikov teach the one or more non-transitory computer-readable media of claim 18.
Although Kim, in view of Dally, De’souza, Weiss, and Supikov substantially teach the claimed invention, Kim, in view of Dally, De’souza, Weiss, and Supikov do not explicitly teach wherein the neural network has been trained to determine the absolute value of the weight.
In the same field, analogous art Frazier teaches wherein the neural network has been trained to determine the absolute value of the weight (see, paragraph 13, “the trained neural network comprises, at least, a dense layer optimized to predict the opportunity score to generate a sum of absolute values of edge weights” [i.e., the neural network has been trained/the trained neural network to determine the absolute value of the weight/generate a sum of absolute values of edge weights]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, Weiss, and Supikov to incorporate the teachings of Frazier so that the trained neural network to generate a sum of absolute values of edge weights (see Frazier e.g., paragraph 13). Doing so would have allowed Kim, in view of Dally, De’souza, Weiss, and Supikov to use Frazier ‘s optimized to predict the opportunity score, as suggested by Frazier (see Frazier, paragraph 13).
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, in view of Dally, De’souza, Weiss, and Supikov as applied to claims 8 and 18 above and further in view of Choi et al. (U.S. Publication No. 20190138882, hereinafter “Choi”).
Regarding claim 10, as discussed above, Kim, in view of Dally, De’souza, Weiss, and Supikov teach the method of claim 8.
Although Kim, in view of Dally, De’souza, Weiss, and Supikov substantially teach the claimed invention, Kim, in view of Dally, De’souza, Weiss, and Supikov do not explicitly teach determining the second activation threshold after adding the weight pruning operation to the layer.
In the same field, analogous art Choi teaches determining the second activation threshold after adding the weight pruning operation to the layer (see, paragraph 81, “for a target weight pruning rate r.sub.l in each layer l, the threshold θ.sub.l may be obtained” [i.e., (Adding weight pruning to the layer to determine the second activation threshold/target weight pruning in each layer, the threshold may be obtained)]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, Weiss, and Supikov to incorporate the teachings of Choi so that target weight pruning in each layer, the threshold may be obtained (see Choi e.g., paragraph 81). Doing so would have allowed Kim, in view of Dally, De’souza, Weiss, and Supikov to use Choi ‘s determined threshold for optimization and quantization, as suggested by Choi (see Choi, paragraph 81).
Regarding claim 20, as discussed above, Kim, in view of Dally, De’souza, Weiss, and Supikov teach the method of claim 18.
Although Kim, in view of Dally, De’souza, Weiss, and Supikov substantially teach the claimed invention, Kim, in view of Dally, De’souza, Weiss, and Supikov do not explicitly teach determining the second activation threshold after adding the weight pruning operation to the layer.
In the same field, analogous art Choi teaches determining the second activation threshold after adding the weight pruning operation to the layer (see, paragraph 81, “for a target weight pruning rate r.sub.l in each layer l, the threshold θ.sub.l may be obtained” [i.e., (Adding weight pruning to the layer to determine the second activation threshold/target weight pruning in each layer, the threshold may be obtained)]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kim, in view of Dally, De’souza, Weiss, and Supikov to incorporate the teachings of Choi so that target weight pruning in each layer, the threshold may be obtained (see Choi e.g., paragraph 81). Doing so would have allowed Kim, in view of Dally, De’souza, Weiss, and Supikov to use Choi ‘s determined threshold for optimization and quantization, as suggested by Choi (see Choi, paragraph 81).
Conclusion
The prior art made of record, listed on form PTO-892, and not relied upon, is considered pertinent to applicant's disclosure.
The references listed on form PTO-892 are all generally related to techniques, methods and systems for pruning tensors and/or neural networks and comparing network accuracy and/or performance to a threshold for purposes of improving performance and efficiency.
For example, LEE; Dongsoo et al. (U.S. Publication No. 20200234131, hereinafter “Lee”) discloses “wherein the second pruning index matrix indicates for each element of the plurality of second elements, whether each element of the plurality of second elements has been pruned, prune, based on a third threshold, each of a plurality of third elements included in the third matrix” (see e.g., paragraph 16).
Also, for example, Seibold; Robin et al. (U.S. Publication No. 20180181867, hereinafter “Seibold”) discloses “Removing a neuron from a convolutional layer means skipping the dot product between one matrix row and one matrix column, which is the same as skipping one convolution. The removal of neurons will be discussed in further detail in the Detailed Specification below. Determining which neurons can be removed without heavily affecting the accuracy of the artificial neural network can be done by analyzing the neurons during the training/test phase” (see e.g., paragraph 3).
Further, for example, Khan; Aftab et al. (U.S. Publication No. 20230061725, hereinafter “Khan”) discloses “when trained on sets of text 302 of at least the threshold clean score, the machine learning algorithm 136 demonstrates at least a minimum accuracy level. In some embodiments, comparison 310 may include comparisons to more than one threshold.” (see e.g., paragraph 49).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN PENG whose telephone number is (571)270-0897. The examiner can normally be reached Monday - Friday 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEVEN PENG/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights