DETAILED ACTION
Claims 1-21 have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 U.S.C. § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The invention, as taught in Claims 1-21, is directed to “mental steps” and “mathematical steps” without significantly more.
The claims recite:
• determining a sensitivity metric value for each of the weights in the machine learning model
• sensitivity metric value indicating influence of each of the weights on an output of the machine learning model
• modifying a first predetermined number or percentage of sensitivity metric values
• selecting a second predetermined number or percentage of the weights as first weights for pruning
• comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights
• first weights pruned to generate a first updated machine learning model with a first sparsity of weights
Claim 1
Step 1 inquiry: Does this claim fall within a statutory category?
The preamble of the claim recites “1. A computer-implemented method, comprising…” Therefore, it is a “method” (or “process”), which is a statutory category of invention. Therefore, the answer to the inquiry is: “YES.”
Step 2A (Prong One) inquiry:
Are there limitations in Claim 1 that recite abstract ideas?
YES. The following limitations in Claim 1 recite abstract ideas that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG. Specifically, they are “mental steps” and “mathematical steps”:
• determining a sensitivity metric value for each of the weights in the machine learning model
• sensitivity metric value indicating influence of each of the weights on an output of the machine learning model
• modifying a first predetermined number or percentage of sensitivity metric values
• selecting a second predetermined number or percentage of the weights as first weights for pruning
• comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights
• first weights pruned to generate a first updated machine learning model with a first sparsity of weights
Step 2A (Prong Two) inquiry:
Are there additional elements or a combination of elements in the claim that apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
This “computer” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(g) recites:
2106.05(g) Insignificant Extra-Solution Activity [R-10.2019]
Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more in Step 2B is whether the additional elements add more than insignificant extra-solution activity to the judicial exception. The term “extra-solution activity” can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity.
This “receiving weights of a plurality of layers of a machine learning model trained using first training data” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level. M.P.E.P. § 2106.05 (f)(2) recites in part:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
This “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
The answer to the inquiry is “NO”, no additional elements integrate the claimed abstract idea into a practical application.
Step 2B inquiry:
Does the claim provide an inventive concept, i.e., does the claim recite additional element(s) or a combination of elements that amount to significantly more than the judicial exception in the claim?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2016.05(f) recites:
2106.05(f) Mere Instructions To Apply An Exception [R-10.2019]
Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words “apply it” (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do “‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’”. Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984 (warning against a § 101 analysis that turns on “the draftsman’s art”).
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0041] recites:
[0041] CPU 102 may be a general-purpose processor using any appropriate architecture. CPU 102 retrieves and executes computer code including instructions, when executed, may cause CPU 102 or another processor, individually or in combination, to perform certain actions or processes that are described in this disclosure. Instructions can be any directions, commands, or orders that may be stored in different forms, such as equipment-readable instructions, programming instructions including source code, and other communication signals and orders. Instructions may be used in a general sense and are not limited to machine-readable codes. CPU 102 may be used to compile the instructions and also determine which processors may be used to perform certain tasks based on the commands in the instructions. For example, certain machine learning computations may be more efficiently performed using AI accelerator 104 while other parallel computations may be better to be processed using GPU 106.
The “computer” is well-understood, routine, and conventional.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
Further, M.P.E.P. § 2106.05(d)(I)(2) recites in part:
2. A factual determination is required to support a conclusion that an additional element (or combination of additional elements) is well-understood, routine, conventional activity. Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018). However, this does not mean that a prior art search is necessary to resolve this inquiry. Instead, examiners should rely on what the courts have recognized, or those in the art would recognize, as elements that are well-understood, routine, conventional activity in the relevant field when making the required determination. For example, in many instances, the specification of the application may indicate that additional elements are well-known or conventional. See, e.g., Intellectual Ventures v. Symantec, 838 F.3d at 1317; 120 USPQ2d at 1359 ("The written description is particularly useful in determining what is well-known or conventional"); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015) (relying on specification’s description of additional elements as "well-known", "common" and "conventional"); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 614, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (Specification described additional elements as "either performing basic computer functions such as sending and receiving data, or performing functions ‘known’ in the art.").
Merely using the conventional computer to receive data is well known, understood, and conventional. Thus, it adds nothing significantly more to the judicial exception.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level.
Further, since the “training the machine learning model” is well understood, routine and conventional, simply using the training of a generic model with unspecified data to produce a result is not eligible. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0048] recites:
[0048] Machine learning models 140 may include different types of algorithms for making inferences based on the training of the models. Examples of machine learning models 140 include regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional network networks (CNNs), recurrent network networks (RNNs), autoencoders, long short term memory (LSTM), reinforcement learning (RL) models, transformers, conformers, and spiking neural networks (SNNs). Some of the machine learning models may include a sparse network structure whose detail will be further discussed with reference to FIG. 2B through 2D. A machine learning model 140 may be an independent model that is run by a processor. A machine learning model 140 may also be part of a software application 130. Machine learning models 140 may perform various tasks.
Therefore, “training the machine learning model” is well-understood, routine, and conventional. Simply using the “training the machine learning model” to produce a result is not eligible.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
Therefore, the answer to the inquiry is “NO”, no additional elements provide an inventive concept that is significantly more than the claimed abstract ideas the claimed abstract idea into a practical application.
Claim 1 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 2
Claim 2 recites:
2. The method of claim 1, further comprising:
determining a sensitivity metric value for each of weights in the first updated machine learning model;
for each subset of weights in a layer of the first updated machine learning model, modifying a second predetermined number or percentage of sensitivity metric values;
across the plurality of layers of the first updated machine learning model, selecting a third predetermined number or percentage the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and
training the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.
Applicant’s Claim 2 merely teaches determination of a mathematical value (i.e., mental steps), modification of mathematical values (i.e., mathematical or mental steps), selection of weights (i.e., mental steps), and comparisons (i.e., mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 2 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 3
Claim 3 recites:
3. The method of claim 2, further comprising:
generating a first mask representing an array with entries corresponding to the weights of the machine learning model;
setting entries of the first mask corresponding to the first weights to zero, where the first mask is applied to the machine learning model for generating the first updated machine learning model;
generating a second mask representing an array with entries corresponding to the weights of the first updated machine learning model; and
setting entries of the second mask corresponding to the second weights to zero, where the second mask is applied to the first updated machine learning model for generating the second updated machine learning model.
Applicant’s Claim 3 merely teaches the generation of mathematical arrays and setting/selecting of mathematical data. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 3 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 4
Claim 4 recites:
4. The method of claim 2, further comprising:
generating a first consolidated tensor concatenating the weights in the machine learning model, the sensitivity metric value of each of the weights in the machine learning model determined by processing the first consolidated tensor; and
generating a second consolidated tensor concatenating the weights in the first updated machine learning model, the sensitivity metric value of each of the weights in the first updated machine learning model determined by processing the second consolidated tensor.
Applicant’s Claim 4 merely teaches generating mathematical tensors and calculating sensitivity metric values. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 4 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 5
Claim 5 recites:
5. The method of claim 2, wherein the training of the machine learning model with the selected weights is performed using second training data that is part of the first training data, and wherein the training of the first updated machine learning model is performed using third training data that is part of the first training data.
Applicant’s Claim 5 merely teaches the selection of training data and training a generic model. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 5 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 6
Claim 6 recites:
6. The method of claim 1, wherein predetermined rules are applied to select the first predetermined number or percentage of the sensitive values, wherein the predetermined rules indicate that sensitivity metric values of higher values are more likely to be modified relative to sensitivity metric values of lower values, and wherein modifying of the first predetermined number or percentage of the sensitivity metric values comprises increasing the first predetermined number or percentage of the sensitivity metric values by a predetermined value.
Applicant’s Claim 6 merely teaches the application of predetermined rules (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 6 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 7
Claim 7 recites:
7. The method of claim 6, wherein the predetermined rules are associated with patterns of weights suitable for accelerated processing by a hardware circuit.
Applicant’s Claim 7 merely teaches predetermined rules (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 7 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 8
Claim 8 recites:
8. The method of claim 1, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.
Applicant’s Claim 8 merely teaches the calculation of metric values from weights and gradients (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 8 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 9
Claim 9 recites:
9. The method of claim 1, further comprising deploying the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.
Applicant’s Claim 9 merely teaches generic deployment of software. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 9 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 10
Step 1 inquiry: Does this claim fall within a statutory category?
The preamble of the claim recites “10. A non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to…” Therefore, it is a “non-transitory storage medium” (or “product of manufacture”), which is a statutory category of invention. Therefore, the answer to the inquiry is: “YES.”
Step 2A (Prong One) inquiry:
Are there limitations in Claim 10 that recite abstract ideas?
YES. The following limitations in Claim 10 recite abstract ideas that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG. Specifically, they are “mental steps” and “mathematical steps”:
• determining a sensitivity metric value for each of the weights in the machine learning model
• sensitivity metric value indicating influence of each of the weights on an output of the machine learning model
• modifying a first predetermined number or percentage of sensitivity metric values
• selecting a second predetermined number or percentage of the weights as first weights for pruning
• comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights
• first weights pruned to generate a first updated machine learning model with a first sparsity of weights
Step 2A (Prong Two) inquiry:
Are there additional elements or a combination of elements in the claim that apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
This “computer” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(g) recites:
2106.05(g) Insignificant Extra-Solution Activity [R-10.2019]
Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more in Step 2B is whether the additional elements add more than insignificant extra-solution activity to the judicial exception. The term “extra-solution activity” can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity.
This “receiving weights of a plurality of layers of a machine learning model trained using first training data” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level. M.P.E.P. § 2106.05 (f)(2) recites in part:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
This “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
The answer to the inquiry is “NO”, no additional elements integrate the claimed abstract idea into a practical application.
Step 2B inquiry:
Does the claim provide an inventive concept, i.e., does the claim recite additional element(s) or a combination of elements that amount to significantly more than the judicial exception in the claim?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2016.05(f) recites:
2106.05(f) Mere Instructions To Apply An Exception [R-10.2019]
Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words “apply it” (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do “‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’”. Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984 (warning against a § 101 analysis that turns on “the draftsman’s art”).
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0041] recites:
[0041] CPU 102 may be a general-purpose processor using any appropriate architecture. CPU 102 retrieves and executes computer code including instructions, when executed, may cause CPU 102 or another processor, individually or in combination, to perform certain actions or processes that are described in this disclosure. Instructions can be any directions, commands, or orders that may be stored in different forms, such as equipment-readable instructions, programming instructions including source code, and other communication signals and orders. Instructions may be used in a general sense and are not limited to machine-readable codes. CPU 102 may be used to compile the instructions and also determine which processors may be used to perform certain tasks based on the commands in the instructions. For example, certain machine learning computations may be more efficiently performed using AI accelerator 104 while other parallel computations may be better to be processed using GPU 106.
The “computer” is well-understood, routine, and conventional.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
Further, M.P.E.P. § 2106.05(d)(I)(2) recites in part:
2. A factual determination is required to support a conclusion that an additional element (or combination of additional elements) is well-understood, routine, conventional activity. Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018). However, this does not mean that a prior art search is necessary to resolve this inquiry. Instead, examiners should rely on what the courts have recognized, or those in the art would recognize, as elements that are well-understood, routine, conventional activity in the relevant field when making the required determination. For example, in many instances, the specification of the application may indicate that additional elements are well-known or conventional. See, e.g., Intellectual Ventures v. Symantec, 838 F.3d at 1317; 120 USPQ2d at 1359 ("The written description is particularly useful in determining what is well-known or conventional"); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015) (relying on specification’s description of additional elements as "well-known", "common" and "conventional"); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 614, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (Specification described additional elements as "either performing basic computer functions such as sending and receiving data, or performing functions ‘known’ in the art.").
Merely using the conventional computer to receive data is well known, understood, and conventional. Thus, it adds nothing significantly more to the judicial exception.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level.
Further, since the “training the machine learning model” is well understood, routine and conventional, simply using the training of a generic model with unspecified data to produce a result is not eligible. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0048] recites:
[0048] Machine learning models 140 may include different types of algorithms for making inferences based on the training of the models. Examples of machine learning models 140 include regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional network networks (CNNs), recurrent network networks (RNNs), autoencoders, long short term memory (LSTM), reinforcement learning (RL) models, transformers, conformers, and spiking neural networks (SNNs). Some of the machine learning models may include a sparse network structure whose detail will be further discussed with reference to FIG. 2B through 2D. A machine learning model 140 may be an independent model that is run by a processor. A machine learning model 140 may also be part of a software application 130. Machine learning models 140 may perform various tasks.
Therefore, “training the machine learning model” is well-understood, routine, and conventional. Simply using the “training the machine learning model” to produce a result is not eligible.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
Therefore, the answer to the inquiry is “NO”, no additional elements provide an inventive concept that is significantly more than the claimed abstract ideas the claimed abstract idea into a practical application.
Claim 10 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 11
Claim 11 recites:
11. The non-transitory storage medium of claim 10, further storing instructions that cause the processor to:
determine a sensitivity metric value for each of weights in the first updated machine learning model;
for each subset of weights in a layer of the first updated machine learning model, modify a second predetermined number or percentage of sensitivity metric values;
across the plurality of layers of the first updated machine learning model, select a third predetermined number or percentage the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and
train the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.
Applicant’s Claim 11 merely teaches determination of a mathematical value (i.e., mental steps), modification of mathematical values (i.e., mathematical or mental steps), selection of weights (i.e., mental steps), and comparisons (i.e., mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 11 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 12
Claim 12 recites:
12. The non-transitory storage medium of claim 11, further storing instructions that cause the processor to:
generate a first mask representing an array with entries corresponding to the weights of the machine learning model;
set entries of the first mask corresponding to the first weights to zero, where the first mask is applied to the machine learning model for generating the first updated machine learning model;
generate a second mask representing an array with entries corresponding to the weights of the first updated machine learning model; and
set entries of the second mask corresponding to the second weights to zero, where the second mask is applied to the first updated machine learning model for generating the second updated machine learning model.
Applicant’s Claim 12 merely teaches the generation of mathematical arrays and setting/selecting of mathematical data. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 12 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 13
Claim 13 recites:
13. The non-transitory storage medium of claim 11, further storing instructions that cause the processor to:
generate a first consolidated tensor concatenating the weights in the machine learning model, the sensitivity metric value of each of the weights in the machine learning model determined by processing the first consolidated tensor; and
generate a second consolidated tensor concatenating the weights in the first updated machine learning model, the sensitivity metric value of each of the weights in the first updated machine learning model determined by processing the second consolidated tensor.
Applicant’s Claim 13 merely teaches generating mathematical tensors and calculating sensitivity metric values. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 13 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 14
Claim 14 recites:
14. The non-transitory storage medium of claim 11, wherein the instructions to train the machine learning model with the selected weights use second training data that is part of the first training data, and wherein the instructions to train the first updated machine learning model uses third training data that is part of the first training data.
Applicant’s Claim 14 merely teaches the selection of training data (i.e., mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 14 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 15
Claim 15 recites:
15. The non-transitory storage medium of claim 10, wherein predetermined rules are applied to select the first predetermined number or percentage of the sensitive values, wherein the predetermined rules indicate that sensitivity metric values of higher values are more likely to be modified relative to sensitivity metric values of lower values, and wherein modifying of the first predetermined number or percentage of the sensitivity metric values comprises increasing the first predetermined number or percentage of the sensitivity metric values by a predetermined value.
Applicant’s Claim 15 merely teaches predetermined rules (i.e., mathematical or mental steps), modifying values (i.e., mathematical or mental steps), and increasing values (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 15 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 16
Claim 16 recites:
16. The non-transitory storage medium of claim 15, wherein the predetermined rules are associated with patterns of weights suitable for accelerated processing by a hardware circuit.
Applicant’s Claim 16 merely teaches predetermined rules (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 16 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 17
Claim 17 recites:
17. The non-transitory storage medium of claim 10, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.
Applicant’s Claim 17 merely teaches the basis of a mathematical value (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 17 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 18
Claim 18 recites:
18. The non-transitory storage medium of claim 10, further storing instructions that cause the processor to deploy the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.
Applicant’s Claim 18 merely teaches generic deployment of software. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 18 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 19
Step 1 inquiry: Does this claim fall within a statutory category?
The preamble of the claim recites “19. A computer-implemented method, comprising…” Therefore, it is a “method” (or “process”), which is a statutory category of invention. Therefore, the answer to the inquiry is: “YES.”
Step 2A (Prong One) inquiry:
Are there limitations in Claim 19 that recite abstract ideas?
YES. The following limitations in Claim 19 recite abstract ideas that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG. Specifically, they are “mental steps” and “mathematical steps”:
• determining a sensitivity metric value for each of the weights in the current machine learning model, the sensitivity metric value indicating influence of each of the weights on an output of the current machine learning model
• sparsifying the weights of the machine learning model by selectively zeroing the weights with lowest sensitivity metric values to generate an intermediate machine learning model
• determining if the updated machine learning model satisfies a termination condition
• setting the updated machine learning model as a sparsified machine learning model
• setting the updated machine learning model as the current machine learning model
Step 2A (Prong Two) inquiry:
Are there additional elements or a combination of elements in the claim that apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data”
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model”
(4) A “repeating (a) through (g)”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
This “computer” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(g) recites:
2106.05(g) Insignificant Extra-Solution Activity [R-10.2019]
Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more in Step 2B is whether the additional elements add more than insignificant extra-solution activity to the judicial exception. The term “extra-solution activity” can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity.
This “receiving weights of a plurality of layers of a current machine learning model trained using first training data” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model” is a broad term which is described at a high level. M.P.E.P. § 2106.05 (f)(2) recites in part:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
This “training the intermediate machine learning model using second training data to generate an updated machine learning model” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(4) A “repeating (a) through (g)” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
***
ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) (“The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.”);
This “repeating (a) through (g)” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
The answer to the inquiry is “NO”, no additional elements integrate the claimed abstract idea into a practical application.
Step 2B inquiry:
Does the claim provide an inventive concept, i.e., does the claim recite additional element(s) or a combination of elements that amount to significantly more than the judicial exception in the claim?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data”
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model”
(4) A “repeating (a) through (g)”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2016.05(f) recites:
2106.05(f) Mere Instructions To Apply An Exception [R-10.2019]
Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words “apply it” (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do “‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’”. Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984 (warning against a § 101 analysis that turns on “the draftsman’s art”).
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0041] recites:
[0041] CPU 102 may be a general-purpose processor using any appropriate architecture. CPU 102 retrieves and executes computer code including instructions, when executed, may cause CPU 102 or another processor, individually or in combination, to perform certain actions or processes that are described in this disclosure. Instructions can be any directions, commands, or orders that may be stored in different forms, such as equipment-readable instructions, programming instructions including source code, and other communication signals and orders. Instructions may be used in a general sense and are not limited to machine-readable codes. CPU 102 may be used to compile the instructions and also determine which processors may be used to perform certain tasks based on the commands in the instructions. For example, certain machine learning computations may be more efficiently performed using AI accelerator 104 while other parallel computations may be better to be processed using GPU 106.
The “computer” is well-understood, routine, and conventional.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
Further, M.P.E.P. § 2106.05(d)(I)(2) recites in part:
2. A factual determination is required to support a conclusion that an additional element (or combination of additional elements) is well-understood, routine, conventional activity. Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018). However, this does not mean that a prior art search is necessary to resolve this inquiry. Instead, examiners should rely on what the courts have recognized, or those in the art would recognize, as elements that are well-understood, routine, conventional activity in the relevant field when making the required determination. For example, in many instances, the specification of the application may indicate that additional elements are well-known or conventional. See, e.g., Intellectual Ventures v. Symantec, 838 F.3d at 1317; 120 USPQ2d at 1359 ("The written description is particularly useful in determining what is well-known or conventional"); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015) (relying on specification’s description of additional elements as "well-known", "common" and "conventional"); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 614, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (Specification described additional elements as "either performing basic computer functions such as sending and receiving data, or performing functions ‘known’ in the art.").
Merely using the conventional computer to receive data is well known, understood, and conventional. Thus, it adds nothing significantly more to the judicial exception.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model” is a broad term which is described at a high level.
Further, since the “training” is well-understood, routine and conventional, simply using the training to produce a result is not eligible. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0048] recites:
[0048] Machine learning models 140 may include different types of algorithms for making inferences based on the training of the models. Examples of machine learning models 140 include regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional network networks (CNNs), recurrent network networks (RNNs), autoencoders, long short term memory (LSTM), reinforcement learning (RL) models, transformers, conformers, and spiking neural networks (SNNs). Some of the machine learning models may include a sparse network structure whose detail will be further discussed with reference to FIG. 2B through 2D. A machine learning model 140 may be an independent model that is run by a processor. A machine learning model 140 may also be part of a software application 130. Machine learning models 140 may perform various tasks.
Therefore, the claimed “training” is well-understood, routine and conventional. Simply using the “training” to produce a result is not eligible.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(4) A “repeating (a) through (g)” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
***
ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) (“The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.”);
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
Therefore, the answer to the inquiry is “NO”, no additional elements provide an inventive concept that is significantly more than the claimed abstract ideas the claimed abstract idea into a practical application.
Claim 19 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 20
Claim 20 recites:
20. The method of claim 19, wherein (c) sparsifying the weights comprises:
(c1) for each subset of weights in a layer of the current machine learning model, selecting a first predetermined number or percentage of weights with highest sensitivity metric values;
(c2) increasing sensitivity metric values of the selected weights;
(c3) selecting a second predetermined number of percentage weights in the machine learning model with lowest sensitivity metric values as weights to be pruned; and
(c4) zeroing the weights to be pruned to generate the intermediate machine learning model.
Applicant’s Claim 20 merely teaches merely selecting weights (i.e., mental steps), increasing mathematical values (i.e., mathematical or mental steps), selecting a number of mathematical weights (i.e., mental steps), and zeroing weighs (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 20 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 21
Claim 21 recites:
21. A non-transitory computer readable storage medium storing a sparse machine learning model generated by a method comprising:
(a) receiving weights of a plurality of layers of a current machine learning model trained using first training data;
(b) determining a sensitivity metric value for each of the weights in the current machine learning model, the sensitivity metric indicating influence of each of the weights on an output of the current machine learning model;
(c) sparsifying the weights of the machine learning model by selectively zeroing the weights with lowest sensitivity metric values to generate an intermediate machine learning model;
(d) training the intermediate machine learning model using second training data to generate an updated machine learning model;
(e) determining if the updated machine learning model satisfies a termination condition;
(f) responsive to determining that the termination condition is satisfied, setting the updated machine learning model as the sparse machine learning model; and
(g) responsive to determining that the termination condition is not satisfied, setting the updated machine learning model as the current machine learning model and repeating (a) through (g).
Applicant’s Claim 21 merely teaches generically receiving “weight” data, “determining a sensitivity metric value” (i.e., mathematical or mental steps), “zeroing” weight data (i.e., mathematical or mental steps), training a generic model, “determining” whether the training should stop (i.e., mental steps), and setting/selecting the model (i.e., weights) as the “model”. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 21 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim Rejections - 35 U.S.C. § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims XXXXXXXXXXXX are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Guo, et al., Sensitivity Pruner: Filter-Level Compression Algorithm for Deep Neural Networks, Pattern Recognition 140 (2023) 109508, 10 MAR 2023, pp. 1-13, in its entirety. Specifically:
Claim 1
Claim 1’s “receiving weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 1’s “determining a sensitivity metric value for each of the weights in the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 1’s “the sensitivity metric value indicating influence of each of the weights on an output of the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 1’s “for each subset of weights in a layer of the machine learning model, modifying a first predetermined number or percentage of sensitivity metric values” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 1’s “across the plurality of layers of the machine learning model, selecting a second predetermined number or percentage of the weights as first weights for pruning by comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights” is anticipated by Guo, et al., page 3, right column, next to last full paragraph, where it recites:
To generate the pruning strategy, in one of the networks, the indicator matrix, initiated with all 1s, is applied to all layers to be pruned. The gradient of this matrix is regarded as the influence matrix, denoted as “Infl” in Fig. 2 . The influence matrix is then summed over the output channel so each channel possesses an importance score. Then, the importance score for all channels to be pruned undergoes a sorting process, and a threshold score defined by a preset compression rate is selected. For the current layer, channels with scores larger than the threshold will be retained; otherwise, they will be pruned. This indicator vector is regarded as the single-shot strategy, which will be the ground truth for the current epoch.
Claim 1’s “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 2
Claim 2’s “determining a sensitivity metric value for each of weights in the first updated machine learning model;” is anticipated by Guo, et al., page 4, right column, second full column, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml. The sensitivity of weight matrix Wl, denoted as the influence matrix Infl , can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 2’s “for each subset of weights in a layer of the first updated machine learning model, modifying a second predetermined number or percentage of sensitivity metric values;” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 2’s “across the plurality of layers of the first updated machine learning model, selecting a third predetermined number or percentage the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 2’s “training the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 5
Claim 5’s “5. The method of claim 2, wherein the training of the machine learning model with the selected weights is performed using second training data that is part of the first training data, and wherein the training of the first updated machine learning model is performed using third training data that is part of the first training data.” Is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 8
Claim 8’s “8. The method of claim 1, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.” is anticipated by Guo, et al., page 2, left column, first bullet point, where it recites:
We integrate the sensitivity measure from SNIP into the “training while fine-tuning” framework to form a more powerful pruning strategy by adapting the unstructured pruning measure from SNIP to allow filter-level compression. In practice, the sensitivity score can be easily computed as the gradient of the connection mask applied to the weight matrix. Independent of the model structure, the sensitivity score can be applied to most neural networks for pruning purposes.
Claim 9
Claim 9’s “9. The method of claim 1, further comprising deploying the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.” is anticipated by Guo, et al., page 6, left column, last full paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradient, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block. To ensure such addition, the last layer of blocks with residual connection ahead is not pruned. In MobileNetV2, point-wise convolution is implemented using the “group” parameter in the PyTorch framework.
Claim 10
Claim 10’s “receive weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 10’s “determine a sensitivity metric value for each of the weights in the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 10’s “the sensitivity metric value indicating influence of each of the weights on an output of the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 10’s “for each subset of weights in a layer of the machine learning model, modifying a first predetermined number or percentage of sensitivity metric values” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 10’s “across the plurality of layers of the machine learning model, selecting a second predetermined number or percentage of the weights as first weights for pruning by comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights” is anticipated by Guo, et al., page 3, right column, next to last full paragraph, where it recites:
To generate the pruning strategy, in one of the networks, the indicator matrix, initiated with all 1s, is applied to all layers to be pruned. The gradient of this matrix is regarded as the influence matrix, denoted as “Infl” in Fig. 2 . The influence matrix is then summed over the output channel so each channel possesses an importance score. Then, the importance score for all channels to be pruned undergoes a sorting process, and a threshold score defined by a preset compression rate is selected. For the current layer, channels with scores larger than the threshold will be retained; otherwise, they will be pruned. This indicator vector is regarded as the single-shot strategy, which will be the ground truth for the current epoch.
Claim 10’s “train the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 11
Claim 11’s “determine a sensitivity metric value for each of weights in the first updated machine learning model;” is anticipated by Guo, et al., page 4, right column, second full column, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml. The sensitivity of weight matrix Wl, denoted as the influence matrix Infl , can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 11’s “for each subset of weights in a layer of the first updated machine learning model, modify a second predetermined number or percentage of sensitivity metric values;” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 11’s “across the plurality of layers of the first updated machine learning model, select a third predetermined number or percentage the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 11’s “train the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 14
Claim 14’s “14. The non-transitory storage medium of claim 11, wherein the instructions to train the machine learning model with the selected weights use second training data that is part of the first training data, and wherein the instructions to train the first updated machine learning model uses third training data that is part of the first training data.” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 17
Claim 17’s “17. The non-transitory storage medium of claim 10, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.” is anticipated by Guo, et al., page 2, left column, first bullet point, where it recites:
We integrate the sensitivity measure from SNIP into the “training while fine-tuning” framework to form a more powerful pruning strategy by adapting the unstructured pruning measure from SNIP to allow filter-level compression. In practice, the sensitivity score can be easily computed as the gradient of the connection mask applied to the weight matrix. Independent of the model structure, the sensitivity score can be applied to most neural networks for pruning purposes.
Claim 18
Claim 18’s “18. The non-transitory storage medium of claim 10, further storing instructions that cause the processor to deploy the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.” is anticipated by Guo, et al., page 6, left column, last full paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradient, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block. To ensure such addition, the last layer of blocks with residual connection ahead is not pruned. In MobileNetV2, point-wise convolution is implemented using the “group” parameter in the PyTorch framework.
Claim 19
Claim 19’s “(a) receiving weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 19’s “(b) determining a sensitivity metric value for each of the weights in the current machine learning model, the sensitivity metric value indicating influence of each of the weights on an output of the current machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 19’s “(c) sparsifying the weights of the machine learning model by selectively zeroing the weights with lowest sensitivity metric values to generate an intermediate machine learning model” is anticipated by Guo, et al., page 4, left column, last partial paragraph, where it recites:
3.3. Sensitivity definition
If we were to prune a connection in weight W l at input channel P , output channel Q, and kernel map position (M, N) , we can simply set the same position in connection matrix to zero, shown as:
Claim 19’s “(d) training the intermediate machine learning model using second training data to generate an updated machine learning model” is anticipated by Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 19’s “(e) determining if the updated machine learning model satisfies a termination condition” Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 19’s “(f) responsive to determining that the termination condition is satisfied, setting the updated machine learning model as the sparse machine learning model; and” is anticipated by Guo, et al., page 6, left column, last partial paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradi- ent, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block.
The model is “set/selected” because it is tested in the prior art.
Claim 19’s “(g) responsive to determining that the termination condition is not satisfied, setting the updated machine learning model as the current machine learning model and repeating (a) through (g).” is anticipated by Guo, et al., page 5, right column, third full paragraph, where it recites:
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Note that in each iteration the model parameters are updated.
Claim 21
Claim 21’s “(a) receiving weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 21’s “(b) determining a sensitivity metric value for each of the weights in the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 21’s “(c) sparsifying the weights of the machine learning model by selectively zeroing the weights with lowest sensitivity metric values to generate an intermediate machine learning model” is anticipated by Guo, et al., page 4, left column, last partial paragraph, where it recites:
3.3. Sensitivity definition
If we were to prune a connection in weight W l at input channel P , output channel Q, and kernel map position (M, N) , we can simply set the same position in connection matrix to zero, shown as:
Claim 21’s “(d) training the intermediate machine learning model using second training data to generate an updated machine learning model” is anticipated by Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 21’s “(e) determining if the updated machine learning model satisfies a termination condition” Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 21’s “(f) responsive to determining that the termination condition is satisfied, setting the updated machine learning model as the sparse machine learning model; and” is anticipated by Guo, et al., page 6, left column, last partial paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradi- ent, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block.
The model is “set/selected” because it is tested in the prior art.
Claim 21’s “(g) responsive to determining that the termination condition is not satisfied, setting the updated machine learning model as the current machine learning model and repeating (a) through (g).” is anticipated by Guo, et al., page 5, right column, third full paragraph, where it recites:
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Note that in each iteration the model parameters are updated.
Reasons for Not Rejecting Claims
Claims 3-4, 6-7, 12-13, 15-16, and 20 are allowed.
Claims 3-4, 6-7, 12-13, 15-16, and 20 are not rejected since when reading the claims in light of the Specification, as per MPEP § 2111.01, none of the references of record, whether taken alone or in combination, discloses or suggests the combination of limitations specified in independent Claims 3 and 12. Specifically, the closest prior art of Guo, et al., Sensitivity Pruner: Filter-Level Compression Algorithm for Deep Neural Networks, Pattern Recognition 140 (2023) 109508, 10 MAR 2023, pp. 1-13 fails to expressly teach:
Claims 3 and 12's "...setting entries of the first mask corresponding to the first weights to zero..."
Claims 3 and 12's "...generating a second mask representing an array..."
Claims 3 and 12's "...setting entries of the second mask corresponding to the second weights to zero..."
Further, none of the references of record, whether taken alone or in combination, discloses or suggests the combination of limitations specified in independent Claims 4 and 13. Specifically, the closest prior art of Guo, et al. fails to expressly teach:
Claims 4 and 13's "...a first consolidated tensor concatenating the weights in the machine learning model..."
Further, none of the references of record, whether taken alone or in combination, discloses or suggests the combination of limitations specified in independent Claims 6 and 15. Specifically, the closest prior art of Guo, et al. fails to expressly teach:
Claims 6 and 15's "...increasing the first predetermined number or percentage of the sensitivity metric values by a predetermined value..."
Further, none of the references of record, whether taken alone or in combination, discloses or suggests the combination of limitations specified in independent Claims 7 and 16. Specifically, the closest prior art of Guo, et al. fails to expressly teach what is incorporated by reference from clams 6 and 15.
Further, none of the references of record, whether taken alone or in combination, discloses or suggests the combination of limitations specified in independent Claim 20. Specifically, the closest prior art of Guo, et al. fails to expressly teach:
Claim 20's "...increasing sensitivity metric values of the selected weights..."
Only to the extent that these limitations (specifically as defined above) are not found in the prior art of record is the present case not rejected over the prior art.
Conclusion
Any inquiries concerning this communication or earlier communications from the examiner should be directed to Wilbert L. Starks, Jr., who may be reached Monday through Friday, between 8:00 a.m. and 5:00 p.m. EST. or via telephone at (571) 272-3691 or email: Wilbert.Starks@uspto.gov.
If you need to send an Official facsimile transmission, please send it to (571) 273-8300.
If attempts to reach the examiner are unsuccessful the Examiner’s Supervisor (SPE), Kakali Chaki, may be reached at (571) 272-3719.
Hand-delivered responses should be delivered to the Receptionist @ (Customer Service Window Randolph Building 401 Dulany Street, Alexandria, VA 22313), located on the first floor of the south side of the Randolph Building.
Finally, information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Moreover, status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have any questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) toll-free @ 1-866-217-9197.
/WILBERT L STARKS/
Primary Examiner, Art Unit 2122
WLS
09 JUN 2026
Prosecution Insights