DETAILED ACTION
Claims 1-21 have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 U.S.C. § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The invention, as taught in Claims 1-21, is directed to “mental steps” and “mathematical steps” without significantly more.
The claims recite:
• determining a sensitivity metric value for each of the weights in the
machine learning model (i.e., mental steps or mathematical steps)
• sensitivity metric value indicating influence of each of the weights on an
output of the machine learning model (i.e., mathematical expression)
• modifying a first predetermined number or percentage of sensitivity
metric values (i.e., mental steps or mathematical steps)
• selecting a second predetermined number or percentage of the weights
as first weights for pruning (i.e., mental steps)
• comparing the sensitivity metric values of the weights, weights
corresponding to the modified sensitivity metric values less likely to be
selected as the first weights (i.e., mental steps)
• first weights pruned to generate a first updated machine learning model
with a first sparsity of weights (i.e., mathematical steps)
• adjusting the first predetermined number or percentage of sensitivity
metric values by a predetermined value (i.e., mathematical steps)
Claim 1
Step 1 inquiry: Does this claim fall within a statutory category?
The preamble of the claim recites “1. A computer-implemented method, comprising…” Therefore, it is a “method” (or “process”), which is a statutory category of invention. Therefore, the answer to the inquiry is: “YES.”
Step 2A (Prong One) inquiry:
Are there limitations in Claim 1 that recite abstract ideas?
YES. The following limitations in Claim 1 recite abstract ideas that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG. Specifically, they are “mental steps” and “mathematical steps”:
• determining a sensitivity metric value for each of the weights in the
machine learning model (i.e., mental steps or mathematical steps)
• sensitivity metric value indicating influence of each of the weights on an
output of the machine learning model (i.e., mathematical expression)
• modifying a first predetermined number or percentage of sensitivity
metric values (i.e., mental steps or mathematical steps)
• selecting a second predetermined number or percentage of the weights
as first weights for pruning (i.e., mental steps)
• comparing the sensitivity metric values of the weights, weights
corresponding to the modified sensitivity metric values less likely to be
selected as the first weights (i.e., mental steps)
• first weights pruned to generate a first updated machine learning model
with a first sparsity of weights (i.e., mathematical steps)
• adjusting the first predetermined number or percentage of sensitivity
metric values by a predetermined value (i.e., mathematical steps)
Step 2A (Prong Two) inquiry:
Are there additional elements or a combination of elements in the claim that apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
This “computer” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(g) recites:
2106.05(g) Insignificant Extra-Solution Activity [R-10.2019]
Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more in Step 2B is whether the additional elements add more than insignificant extra-solution activity to the judicial exception. The term “extra-solution activity” can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity.
This “receiving weights of a plurality of layers of a machine learning model trained using first training data” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level. M.P.E.P. § 2106.05 (f)(2) recites in part:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
This “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
The answer to the inquiry is “NO”, no additional elements integrate the claimed abstract idea into a practical application.
Step 2B inquiry:
Does the claim provide an inventive concept, i.e., does the claim recite additional element(s) or a combination of elements that amount to significantly more than the judicial exception in the claim?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2016.05(f) recites:
2106.05(f) Mere Instructions To Apply An Exception [R-10.2019]
Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words “apply it” (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do “‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’”. Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984 (warning against a § 101 analysis that turns on “the draftsman’s art”).
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0041] recites:
[0041] CPU 102 may be a general-purpose processor using any appropriate architecture. CPU 102 retrieves and executes computer code including instructions, when executed, may cause CPU 102 or another processor, individually or in combination, to perform certain actions or processes that are described in this disclosure. Instructions can be any directions, commands, or orders that may be stored in different forms, such as equipment-readable instructions, programming instructions including source code, and other communication signals and orders. Instructions may be used in a general sense and are not limited to machine-readable codes. CPU 102 may be used to compile the instructions and also determine which processors may be used to perform certain tasks based on the commands in the instructions. For example, certain machine learning computations may be more efficiently performed using AI accelerator 104 while other parallel computations may be better to be processed using GPU 106.
The “computer” is well-understood, routine, and conventional.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
Further, M.P.E.P. § 2106.05(d)(I)(2) recites in part:
2. A factual determination is required to support a conclusion that an additional element (or combination of additional elements) is well-understood, routine, conventional activity. Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018). However, this does not mean that a prior art search is necessary to resolve this inquiry. Instead, examiners should rely on what the courts have recognized, or those in the art would recognize, as elements that are well-understood, routine, conventional activity in the relevant field when making the required determination. For example, in many instances, the specification of the application may indicate that additional elements are well-known or conventional. See, e.g., Intellectual Ventures v. Symantec, 838 F.3d at 1317; 120 USPQ2d at 1359 ("The written description is particularly useful in determining what is well-known or conventional"); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015) (relying on specification’s description of additional elements as "well-known", "common" and "conventional"); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 614, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (Specification described additional elements as "either performing basic computer functions such as sending and receiving data, or performing functions ‘known’ in the art.").
Merely using the conventional computer to receive data is well known, understood, and conventional. Thus, it adds nothing significantly more to the judicial exception.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level.
Further, since the “training the machine learning model” is well understood, routine and conventional, simply using the training of a generic model with unspecified data to produce a result is not eligible. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0048] recites:
[0048] Machine learning models 140 may include different types of algorithms for making inferences based on the training of the models. Examples of machine learning models 140 include regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional network networks (CNNs), recurrent network networks (RNNs), autoencoders, long short term memory (LSTM), reinforcement learning (RL) models, transformers, conformers, and spiking neural networks (SNNs). Some of the machine learning models may include a sparse network structure whose detail will be further discussed with reference to FIG. 2B through 2D. A machine learning model 140 may be an independent model that is run by a processor. A machine learning model 140 may also be part of a software application 130. Machine learning models 140 may perform various tasks.
Therefore, “training the machine learning model” is well-understood, routine, and conventional. Simply using the “training the machine learning model” to produce a result is not eligible.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
Therefore, the answer to the inquiry is “NO”, no additional elements provide an inventive concept that is significantly more than the claimed abstract ideas the claimed abstract idea into a practical application.
Claim 1 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 2
Claim 2 recites:
2. The method of claim 1, further comprising:
determining a sensitivity metric value for each of weights in the first updated machine learning model;
for each subset of weights in a layer of the first updated machine learning model, modifying a second predetermined number or percentage of sensitivity metric values;
across the plurality of layers of the first updated machine learning model, selecting a third predetermined number or percentage of the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and
training the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.
Applicant’s Claim 2 merely teaches determination of a mathematical value (i.e., mental steps), modification of mathematical values (i.e., mathematical or mental steps), selection of weights (i.e., mental steps), and comparisons (i.e., mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 2 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 3
Claim 3 recites:
3. The method of claim 2, further comprising:
generating a first mask representing an array with entries corresponding to the weights of the machine learning model;
setting entries of the first mask corresponding to the first weights to zero, where the first mask is applied to the machine learning model for generating the first updated machine learning model;
generating a second mask representing an array with entries corresponding to the weights of the first updated machine learning model; and
setting entries of the second mask corresponding to the second weights to zero, where the second mask is applied to the first updated machine learning model for generating the second updated machine learning model.
Applicant’s Claim 3 merely teaches the generation of mathematical arrays and setting/selecting of mathematical data. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 3 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 4
Claim 4 recites:
4. The method of claim 2, further comprising:
generating a first consolidated tensor concatenating the weights in the machine learning model, the sensitivity metric value of each of the weights in the machine learning model determined by processing the first consolidated tensor; and
generating a second consolidated tensor concatenating the weights in the first updated machine learning model, the sensitivity metric value of each of the weights in the first updated machine learning model determined by processing the second consolidated tensor.
Applicant’s Claim 4 merely teaches generating mathematical tensors and calculating sensitivity metric values. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 4 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 5
Claim 5 recites:
5. The method of claim 2, wherein the training of the machine learning model with the first weights is performed using second training data that is part of the first training data, and wherein the training of the first updated machine learning model is performed using third training data that is part of the first training data.
Applicant’s Claim 5 merely teaches the selection of training data and training a generic model. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 5 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 6
Claim 6 recites:
6. The method of claim 1, wherein predetermined rules are applied to select the first predetermined number or percentage of the sensitivity metric values, wherein the predetermined rules indicate that sensitivity metric values of higher values are more likely to be modified relative to sensitivity metric values of lower values, and wherein modifying of the first predetermined number or percentage of the sensitivity metric values comprises increasing the first predetermined number or percentage of the sensitivity metric values by a predetermined value.
Applicant’s Claim 6 merely teaches the application of predetermined rules (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 6 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 7
Claim 7 recites:
7. The method of claim 6, wherein the predetermined rules are associated with patterns of weights suitable for accelerated processing by a hardware circuit.
Applicant’s Claim 7 merely teaches predetermined rules (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 7 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 8
Claim 8 recites:
8. The method of claim 1, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.
Applicant’s Claim 8 merely teaches the calculation of metric values from weights and gradients (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 8 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 9
Claim 9 recites:
9. The method of claim 1, further comprising deploying the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.
Applicant’s Claim 9 merely teaches generic deployment of software. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 9 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 10
Step 1 inquiry: Does this claim fall within a statutory category?
The preamble of the claim recites “10. A non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to…” Therefore, it is a “non-transitory storage medium” (or “product of manufacture”), which is a statutory category of invention. Therefore, the answer to the inquiry is: “YES.”
Step 2A (Prong One) inquiry:
Are there limitations in Claim 10 that recite abstract ideas?
YES. The following limitations in Claim 10 recite abstract ideas that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG. Specifically, they are “mental steps” and “mathematical steps”:
• determining a sensitivity metric value for each of the weights in the
machine learning model (i.e., mental steps or mathematical steps)
• sensitivity metric value indicating influence of each of the weights on an
output of the machine learning model (i.e., mathematical expression)
• modifying a first predetermined number or percentage of sensitivity
metric values (i.e., mental steps or mathematical steps)
• selecting a second predetermined number or percentage of the weights
as first weights for pruning (i.e., mental steps)
• comparing the sensitivity metric values of the weights, weights
corresponding to the modified sensitivity metric values less likely to be
selected as the first weights (i.e., mental steps)
• first weights pruned to generate a first updated machine learning model
with a first sparsity of weights (i.e., mathematical steps)
• adjusting the first predetermined number or percentage of sensitivity
metric values by a predetermined value (i.e., mathematical steps)
Step 2A (Prong Two) inquiry:
Are there additional elements or a combination of elements in the claim that apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
This “computer” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(g) recites:
2106.05(g) Insignificant Extra-Solution Activity [R-10.2019]
Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more in Step 2B is whether the additional elements add more than insignificant extra-solution activity to the judicial exception. The term “extra-solution activity” can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity.
This “receiving weights of a plurality of layers of a machine learning model trained using first training data” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level. M.P.E.P. § 2106.05 (f)(2) recites in part:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
This “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
The answer to the inquiry is “NO”, no additional elements integrate the claimed abstract idea into a practical application.
Step 2B inquiry:
Does the claim provide an inventive concept, i.e., does the claim recite additional element(s) or a combination of elements that amount to significantly more than the judicial exception in the claim?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data”
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2016.05(f) recites:
2106.05(f) Mere Instructions To Apply An Exception [R-10.2019]
Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words “apply it” (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do “‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’”. Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984 (warning against a § 101 analysis that turns on “the draftsman’s art”).
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0041] recites:
[0041] CPU 102 may be a general-purpose processor using any appropriate architecture. CPU 102 retrieves and executes computer code including instructions, when executed, may cause CPU 102 or another processor, individually or in combination, to perform certain actions or processes that are described in this disclosure. Instructions can be any directions, commands, or orders that may be stored in different forms, such as equipment-readable instructions, programming instructions including source code, and other communication signals and orders. Instructions may be used in a general sense and are not limited to machine-readable codes. CPU 102 may be used to compile the instructions and also determine which processors may be used to perform certain tasks based on the commands in the instructions. For example, certain machine learning computations may be more efficiently performed using AI accelerator 104 while other parallel computations may be better to be processed using GPU 106.
The “computer” is well-understood, routine, and conventional.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(2) A “receiving weights of a plurality of layers of a machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
Further, M.P.E.P. § 2106.05(d)(I)(2) recites in part:
2. A factual determination is required to support a conclusion that an additional element (or combination of additional elements) is well-understood, routine, conventional activity. Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018). However, this does not mean that a prior art search is necessary to resolve this inquiry. Instead, examiners should rely on what the courts have recognized, or those in the art would recognize, as elements that are well-understood, routine, conventional activity in the relevant field when making the required determination. For example, in many instances, the specification of the application may indicate that additional elements are well-known or conventional. See, e.g., Intellectual Ventures v. Symantec, 838 F.3d at 1317; 120 USPQ2d at 1359 ("The written description is particularly useful in determining what is well-known or conventional"); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015) (relying on specification’s description of additional elements as "well-known", "common" and "conventional"); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 614, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (Specification described additional elements as "either performing basic computer functions such as sending and receiving data, or performing functions ‘known’ in the art.").
Merely using the conventional computer to receive data is well known, understood, and conventional. Thus, it adds nothing significantly more to the judicial exception.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(3) A “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is a broad term which is described at a high level.
Further, since the “training the machine learning model” is well understood, routine and conventional, simply using the training of a generic model with unspecified data to produce a result is not eligible. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0048] recites:
[0048] Machine learning models 140 may include different types of algorithms for making inferences based on the training of the models. Examples of machine learning models 140 include regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional network networks (CNNs), recurrent network networks (RNNs), autoencoders, long short term memory (LSTM), reinforcement learning (RL) models, transformers, conformers, and spiking neural networks (SNNs). Some of the machine learning models may include a sparse network structure whose detail will be further discussed with reference to FIG. 2B through 2D. A machine learning model 140 may be an independent model that is run by a processor. A machine learning model 140 may also be part of a software application 130. Machine learning models 140 may perform various tasks.
Therefore, “training the machine learning model” is well-understood, routine, and conventional. Simply using the “training the machine learning model” to produce a result is not eligible.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
Therefore, the answer to the inquiry is “NO”, no additional elements provide an inventive concept that is significantly more than the claimed abstract ideas the claimed abstract idea into a practical application.
Claim 10 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 11
Claim 11 recites:
11. The non-transitory storage medium of claim 10, further storing instructions that cause the processor to:
determine a sensitivity metric value for each of weights in the first updated machine learning model;
for each subset of weights in a layer of the first updated machine learning model, modify a second predetermined number or percentage of sensitivity metric values;
across the plurality of layers of the first updated machine learning model, select a third predetermined number or percentage of the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and
train the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.
Applicant’s Claim 11 merely teaches determination of a mathematical value (i.e., mental steps), modification of mathematical values (i.e., mathematical or mental steps), selection of weights (i.e., mental steps), and comparisons (i.e., mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 11 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 12
Claim 12 recites:
12. The non-transitory storage medium of claim 11, further storing instructions that cause the processor to:
generate a first mask representing an array with entries corresponding to the weights of the machine learning model;
set entries of the first mask corresponding to the first weights to zero, where the first mask is applied to the machine learning model for generating the first updated machine learning model;
generate a second mask representing an array with entries corresponding to the weights of the first updated machine learning model; and
set entries of the second mask corresponding to the second weights to zero, where the second mask is applied to the first updated machine learning model for generating the second updated machine learning model.
Applicant’s Claim 12 merely teaches the generation of mathematical arrays and setting/selecting of mathematical data. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 12 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 13
Claim 13 recites:
13. The non-transitory storage medium of claim 11, further storing instructions that cause the processor to:
generate a first consolidated tensor concatenating the weights in the machine learning model, the sensitivity metric value of each of the weights in the machine learning model determined by processing the first consolidated tensor; and
generate a second consolidated tensor concatenating the weights in the first updated machine learning model, the sensitivity metric value of each of the weights in the first updated machine learning model determined by processing the second consolidated tensor.
Applicant’s Claim 13 merely teaches generating mathematical tensors and calculating sensitivity metric values. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 13 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 14
Claim 14 recites:
14. The non-transitory storage medium of claim 11, wherein the instructions to train the machine learning model with the first weights use second training data that is part of the first training data, and wherein the instructions to train the first updated machine learning model use third training data that is part of the first training data.
Applicant’s Claim 14 merely teaches the selection of training data (i.e., mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 14 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 15
Claim 15 recites:
15. The non-transitory storage medium of claim 10, wherein predetermined rules are applied to select the first predetermined number or percentage of the sensitivity metric values, wherein the predetermined rules indicate that sensitivity metric values of higher values are more likely to be modified relative to sensitivity metric values of lower values, and wherein modifying of the first predetermined number or percentage of the sensitivity metric values comprises increasing the first predetermined number or percentage of the sensitivity metric values by a predetermined value.
Applicant’s Claim 15 merely teaches predetermined rules (i.e., mathematical or mental steps), modifying values (i.e., mathematical or mental steps), and increasing values (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 15 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 16
Claim 16 recites:
16. The non-transitory storage medium of claim 15, wherein the predetermined rules are associated with patterns of weights suitable for accelerated processing by a hardware circuit.
Applicant’s Claim 16 merely teaches predetermined rules (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 16 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 17
Claim 17 recites:
17. The non-transitory storage medium of claim 10, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.
Applicant’s Claim 17 merely teaches the basis of a mathematical value (i.e., mathematical or mental steps). It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 17 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 18
Claim 18 recites:
18. The non-transitory storage medium of claim 10, further storing instructions that cause the processor to deploy the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.
Applicant’s Claim 18 merely teaches generic deployment of software. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 18 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 19
Step 1 inquiry: Does this claim fall within a statutory category?
The preamble of the claim recites “19. A computer-implemented method, comprising:…” Therefore, it is a “method” (or “process”), which is a statutory category of invention. Therefore, the answer to the inquiry is: “YES.”
Step 2A (Prong One) inquiry:
Are there limitations in Claim 19 that recite abstract ideas?
YES. The following limitations in Claim 19 recite abstract ideas that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG. Specifically, they are “mental steps” and “mathematical steps”:
• determining a sensitivity metric value for each of the weights in the current machine learning model, the sensitivity metric value indicating influence of each of the weights on an output of the current machine learning model (i.e., mental steps)
• sparsifying the weights of the machine learning model (i.e., mathematical steps)
• determining if the updated machine learning model satisfies a termination condition (i.e., mental steps)
• setting the updated machine learning model as a sparsified machine learning model (i.e., mental steps or mathematical steps)
• setting the updated machine learning model as the current machine learning model (i.e., mental steps or mathematical steps)
• (c1) for each subset of weights in a layer of the current machine learning model, selecting a first predetermined number or percentage of weights with highest sensitivity metric values; (i.e., mental steps)
• (c2) increasing sensitivity metric values of the selected weights; (i.e., mental steps or mathematical steps)
• (c3) selecting a second predetermined number of percentage weights in the machine learning model with lowest sensitivity metric values as weights to be pruned; and (i.e., mental steps)
• (c4) zeroing the weights to be pruned to generate the intermediate machine learning model. (i.e., mental steps or mathematical steps)
Step 2A (Prong Two) inquiry:
Are there additional elements or a combination of elements in the claim that apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data”
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model”
(4) A “repeating (a) through (g)”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
This “computer” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(g) recites:
2106.05(g) Insignificant Extra-Solution Activity [R-10.2019]
Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more in Step 2B is whether the additional elements add more than insignificant extra-solution activity to the judicial exception. The term “extra-solution activity” can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity.
This “receiving weights of a plurality of layers of a current machine learning model trained using first training data” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model” is a broad term which is described at a high level. M.P.E.P. § 2106.05 (f)(2) recites in part:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
This “training the intermediate machine learning model using second training data to generate an updated machine learning model” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(4) A “repeating (a) through (g)” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
***
ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) (“The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.”);
This “repeating (a) through (g)” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
The answer to the inquiry is “NO”, no additional elements integrate the claimed abstract idea into a practical application.
Step 2B inquiry:
Does the claim provide an inventive concept, i.e., does the claim recite additional element(s) or a combination of elements that amount to significantly more than the judicial exception in the claim?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data”
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model”
(4) A “repeating (a) through (g)”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2016.05(f) recites:
2106.05(f) Mere Instructions To Apply An Exception [R-10.2019]
Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words “apply it” (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do “‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’”. Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984 (warning against a § 101 analysis that turns on “the draftsman’s art”).
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0041] recites:
[0041] CPU 102 may be a general-purpose processor using any appropriate architecture. CPU 102 retrieves and executes computer code including instructions, when executed, may cause CPU 102 or another processor, individually or in combination, to perform certain actions or processes that are described in this disclosure. Instructions can be any directions, commands, or orders that may be stored in different forms, such as equipment-readable instructions, programming instructions including source code, and other communication signals and orders. Instructions may be used in a general sense and are not limited to machine-readable codes. CPU 102 may be used to compile the instructions and also determine which processors may be used to perform certain tasks based on the commands in the instructions. For example, certain machine learning computations may be more efficiently performed using AI accelerator 104 while other parallel computations may be better to be processed using GPU 106.
The “computer” is well-understood, routine, and conventional.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
Further, M.P.E.P. § 2106.05(d)(I)(2) recites in part:
2. A factual determination is required to support a conclusion that an additional element (or combination of additional elements) is well-understood, routine, conventional activity. Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018). However, this does not mean that a prior art search is necessary to resolve this inquiry. Instead, examiners should rely on what the courts have recognized, or those in the art would recognize, as elements that are well-understood, routine, conventional activity in the relevant field when making the required determination. For example, in many instances, the specification of the application may indicate that additional elements are well-known or conventional. See, e.g., Intellectual Ventures v. Symantec, 838 F.3d at 1317; 120 USPQ2d at 1359 ("The written description is particularly useful in determining what is well-known or conventional"); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015) (relying on specification’s description of additional elements as "well-known", "common" and "conventional"); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 614, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (Specification described additional elements as "either performing basic computer functions such as sending and receiving data, or performing functions ‘known’ in the art.").
Merely using the conventional computer to receive data is well known, understood, and conventional. Thus, it adds nothing significantly more to the judicial exception.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model” is a broad term which is described at a high level.
Further, since the “training” is well-understood, routine and conventional, simply using the training to produce a result is not eligible. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0048] recites:
[0048] Machine learning models 140 may include different types of algorithms for making inferences based on the training of the models. Examples of machine learning models 140 include regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional network networks (CNNs), recurrent network networks (RNNs), autoencoders, long short term memory (LSTM), reinforcement learning (RL) models, transformers, conformers, and spiking neural networks (SNNs). Some of the machine learning models may include a sparse network structure whose detail will be further discussed with reference to FIG. 2B through 2D. A machine learning model 140 may be an independent model that is run by a processor. A machine learning model 140 may also be part of a software application 130. Machine learning models 140 may perform various tasks.
Therefore, the claimed “training” is well-understood, routine and conventional. Simply using the “training” to produce a result is not eligible.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(4) A “repeating (a) through (g)” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
***
ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) (“The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.”);
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
Therefore, the answer to the inquiry is “NO”, no additional elements provide an inventive concept that is significantly more than the claimed abstract ideas the claimed abstract idea into a practical application.
Claim 19 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 21
Step 1 inquiry: Does this claim fall within a statutory category?
The preamble of the claim recites “A non-transitory computer readable storage medium storing A sparse machine learning model generated by a method comprising:…” Therefore, it is a “non-transitory computer readable storage medium” (or “product of manufacture”), which is a statutory category of invention. Therefore, the answer to the inquiry is: “YES.”
Step 2A (Prong One) inquiry:
Are there limitations in Claim 21 that recite abstract ideas?
YES. The following limitations in Claim 21 recite abstract ideas that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG. Specifically, they are “mental steps” and “mathematical steps”:
• determining a sensitivity metric value for each of the weights in the current machine learning model, the sensitivity metric value indicating influence of each of the weights on an output of the current machine learning model (i.e., mental steps)
• sparsifying the weights of the machine learning model (i.e., mathematical steps)
• determining if the updated machine learning model satisfies a termination condition (i.e., mental steps)
• setting the updated machine learning model as a sparsified machine learning model (i.e., mental steps or mathematical steps)
• setting the updated machine learning model as the current machine learning model (i.e., mental steps or mathematical steps)
• (c1) for each subset of weights in a layer of the current machine learning model, selecting a first predetermined number or percentage of weights with highest sensitivity metric values; (i.e., mental steps)
• (c2) increasing sensitivity metric values of the selected weights; (i.e., mental steps or mathematical steps)
• (c3) selecting a second predetermined number of percentage weights in the machine learning model with lowest sensitivity metric values as weights to be pruned; and (i.e., mental steps)
• (c4) zeroing the weights to be pruned to generate the intermediate machine learning model. (i.e., mental steps or mathematical steps)
Step 2A (Prong Two) inquiry:
Are there additional elements or a combination of elements in the claim that apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that it is more than a drafting effort designed to monopolize the exception?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data”
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model”
(4) A “repeating (a) through (g)”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
This “computer” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(g) recites:
2106.05(g) Insignificant Extra-Solution Activity [R-10.2019]
Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more in Step 2B is whether the additional elements add more than insignificant extra-solution activity to the judicial exception. The term “extra-solution activity” can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity.
This “receiving weights of a plurality of layers of a current machine learning model trained using first training data” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model” is a broad term which is described at a high level. M.P.E.P. § 2106.05 (f)(2) recites in part:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
This “training the intermediate machine learning model using second training data to generate an updated machine learning model” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
(4) A “repeating (a) through (g)” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
***
ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) (“The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.”);
This “repeating (a) through (g)” limitation does not integrate the additional element into a practical application and represents “insignificant extra-solution activity”. (See, M.P.E.P. § 2106.05(I)(A)).
The answer to the inquiry is “NO”, no additional elements integrate the claimed abstract idea into a practical application.
Step 2B inquiry:
Does the claim provide an inventive concept, i.e., does the claim recite additional element(s) or a combination of elements that amount to significantly more than the judicial exception in the claim?
Applicant’s claims contain the following “additional elements”:
(1) A “computer”
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data”
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model”
(4) A “repeating (a) through (g)”
(1) A “computer” is a broad term which is described at a high level and includes general purpose computers. M.P.E.P. § 2016.05(f) recites:
2106.05(f) Mere Instructions To Apply An Exception [R-10.2019]
Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words “apply it” (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do “‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’”. Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983. See also 573 U.S. at 224, 110 USPQ2d at 1984 (warning against a § 101 analysis that turns on “the draftsman’s art”).
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0041] recites:
[0041] CPU 102 may be a general-purpose processor using any appropriate architecture. CPU 102 retrieves and executes computer code including instructions, when executed, may cause CPU 102 or another processor, individually or in combination, to perform certain actions or processes that are described in this disclosure. Instructions can be any directions, commands, or orders that may be stored in different forms, such as equipment-readable instructions, programming instructions including source code, and other communication signals and orders. Instructions may be used in a general sense and are not limited to machine-readable codes. CPU 102 may be used to compile the instructions and also determine which processors may be used to perform certain tasks based on the commands in the instructions. For example, certain machine learning computations may be more efficiently performed using AI accelerator 104 while other parallel computations may be better to be processed using GPU 106.
The “computer” is well-understood, routine, and conventional.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(2) A “receiving weights of a plurality of layers of a current machine learning model trained using first training data” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); …
Further, M.P.E.P. § 2106.05(d)(I)(2) recites in part:
2. A factual determination is required to support a conclusion that an additional element (or combination of additional elements) is well-understood, routine, conventional activity. Berkheimer v. HP, Inc., 881 F.3d 1360, 1368, 125 USPQ2d 1649, 1654 (Fed. Cir. 2018). However, this does not mean that a prior art search is necessary to resolve this inquiry. Instead, examiners should rely on what the courts have recognized, or those in the art would recognize, as elements that are well-understood, routine, conventional activity in the relevant field when making the required determination. For example, in many instances, the specification of the application may indicate that additional elements are well-known or conventional. See, e.g., Intellectual Ventures v. Symantec, 838 F.3d at 1317; 120 USPQ2d at 1359 ("The written description is particularly useful in determining what is well-known or conventional"); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015) (relying on specification’s description of additional elements as "well-known", "common" and "conventional"); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 614, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (Specification described additional elements as "either performing basic computer functions such as sending and receiving data, or performing functions ‘known’ in the art.").
Merely using the conventional computer to receive data is well known, understood, and conventional. Thus, it adds nothing significantly more to the judicial exception.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(3) A “training the intermediate machine learning model using second training data to generate an updated machine learning model” is a broad term which is described at a high level.
Further, since the “training” is well-understood, routine and conventional, simply using the training to produce a result is not eligible. M.P.E.P. § 2106.05(f) recites:
For claim limitations that do not amount to more than a recitation of the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners should explain why they do not meaningfully limit the claim in an eligibility rejection. For example, an examiner could explain that implementing an abstract idea on a generic computer, does not integrate the abstract idea into a practical application in Step 2A Prong Two…
Further, M.P.E.P. § 2106.05(f)(2) recites:
(2) Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015). In contrast, a claim that purports to improve computer capabilities or to improve an existing technology may integrate a judicial exception into a practical application or provide significantly more. McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1314-15, 120 USPQ2d 1091, 1101-02 (Fed. Cir. 2016); Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36, 118 USPQ2d 1684, 1688-89 (Fed. Cir. 2016). See MPEP §§ 2106.04(d)(1) and 2106.05(a) for a discussion of improvements to the functioning of a computer or to another technology or technical field.
Applicant's Specification, paragraph [0048] recites:
[0048] Machine learning models 140 may include different types of algorithms for making inferences based on the training of the models. Examples of machine learning models 140 include regression models, random forest models, support vector machines (SVMs) such as kernel SVMs, and artificial neural networks (ANNs) such as convolutional network networks (CNNs), recurrent network networks (RNNs), autoencoders, long short term memory (LSTM), reinforcement learning (RL) models, transformers, conformers, and spiking neural networks (SNNs). Some of the machine learning models may include a sparse network structure whose detail will be further discussed with reference to FIG. 2B through 2D. A machine learning model 140 may be an independent model that is run by a processor. A machine learning model 140 may also be part of a software application 130. Machine learning models 140 may perform various tasks.
Therefore, the claimed “training” is well-understood, routine and conventional. Simply using the “training” to produce a result is not eligible.
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
(4) A “repeating (a) through (g)” is a broad term which is described at a high level. M.P.E.P. § 2106.05(d)(II) recites:
The courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity.
***
ii. Performing repetitive calculations, Flook, 437 U.S. at 594, 198 USPQ2d at 199 (recomputing or readjusting alarm limit values); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) (“The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.”);
Therefore, the claim as a whole does not amount to significantly more than the exception itself (i.e., there is no inventive concept in the claim). (See, M.P.E.P. § 2106.05(II)).
Therefore, the answer to the inquiry is “NO”, no additional elements provide an inventive concept that is significantly more than the claimed abstract ideas the claimed abstract idea into a practical application.
Claim 21 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim 22
Claim 22 recites:
22. (New) The method of claim 1, wherein each subset of weights corresponds to a segment of a layer having a predetermined size, and a predetermined number of weights within each segment are selected as the first weights, forming a partitioned sparsity pattern suitable for accelerated processing by a hardware circuit.
Applicant’s Claim 22 merely teaches a selection of first weights. It does not integrate the abstract idea to a practical application, nor is it anything significantly more than the abstract idea. (See, 2106.05(a)(II).)
Claim 22 is, therefore, NOT ELIGIBLE subject matter under 35 U.S.C. § 101.
Claim Rejections - 35 U.S.C. § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims XXXXXXXXXXXX are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Guo, et al., Sensitivity Pruner: Filter-Level Compression Algorithm for Deep Neural Networks, Pattern Recognition 140 (2023) 109508, 10 MAR 2023, pp. 1-13, in its entirety. Specifically:
Claim 1
Claim 1’s “receiving weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 1’s “determining a sensitivity metric value for each of the weights in the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 1’s “the sensitivity metric value indicating influence of each of the weights on an output of the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 1’s “for each subset of weights in a layer of the machine learning model, modifying a first predetermined number or percentage of sensitivity metric values” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 1’s “across the plurality of layers of the machine learning model, selecting a second predetermined number or percentage of the weights as first weights for pruning by comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights” is anticipated by Guo, et al., page 3, right column, next to last full paragraph, where it recites:
To generate the pruning strategy, in one of the networks, the indicator matrix, initiated with all 1s, is applied to all layers to be pruned. The gradient of this matrix is regarded as the influence matrix, denoted as “Infl” in Fig. 2 . The influence matrix is then summed over the output channel so each channel possesses an importance score. Then, the importance score for all channels to be pruned undergoes a sorting process, and a threshold score defined by a preset compression rate is selected. For the current layer, channels with scores larger than the threshold will be retained; otherwise, they will be pruned. This indicator vector is regarded as the single-shot strategy, which will be the ground truth for the current epoch.
Claim 1’s “training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 2
Claim 2’s “determining a sensitivity metric value for each of weights in the first updated machine learning model;” is anticipated by Guo, et al., page 4, right column, second full column, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml. The sensitivity of weight matrix Wl, denoted as the influence matrix Infl , can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 2’s “for each subset of weights in a layer of the first updated machine learning model, modifying a second predetermined number or percentage of sensitivity metric values;” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 2’s “across the plurality of layers of the first updated machine learning model, selecting a third predetermined number or percentage the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 2’s “training the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 5
Claim 5’s “5. The method of claim 2, wherein the training of the machine learning model with the selected weights is performed using second training data that is part of the first training data, and wherein the training of the first updated machine learning model is performed using third training data that is part of the first training data.” Is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 8
Claim 8’s “8. The method of claim 1, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.” is anticipated by Guo, et al., page 2, left column, first bullet point, where it recites:
We integrate the sensitivity measure from SNIP into the “training while fine-tuning” framework to form a more powerful pruning strategy by adapting the unstructured pruning measure from SNIP to allow filter-level compression. In practice, the sensitivity score can be easily computed as the gradient of the connection mask applied to the weight matrix. Independent of the model structure, the sensitivity score can be applied to most neural networks for pruning purposes.
Claim 9
Claim 9’s “9. The method of claim 1, further comprising deploying the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.” is anticipated by Guo, et al., page 6, left column, last full paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradient, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block. To ensure such addition, the last layer of blocks with residual connection ahead is not pruned. In MobileNetV2, point-wise convolution is implemented using the “group” parameter in the PyTorch framework.
Claim 10
Claim 10’s “receive weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 10’s “determine a sensitivity metric value for each of the weights in the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 10’s “the sensitivity metric value indicating influence of each of the weights on an output of the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 10’s “for each subset of weights in a layer of the machine learning model, modifying a first predetermined number or percentage of sensitivity metric values” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 10’s “across the plurality of layers of the machine learning model, selecting a second predetermined number or percentage of the weights as first weights for pruning by comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights” is anticipated by Guo, et al., page 3, right column, next to last full paragraph, where it recites:
To generate the pruning strategy, in one of the networks, the indicator matrix, initiated with all 1s, is applied to all layers to be pruned. The gradient of this matrix is regarded as the influence matrix, denoted as “Infl” in Fig. 2 . The influence matrix is then summed over the output channel so each channel possesses an importance score. Then, the importance score for all channels to be pruned undergoes a sorting process, and a threshold score defined by a preset compression rate is selected. For the current layer, channels with scores larger than the threshold will be retained; otherwise, they will be pruned. This indicator vector is regarded as the single-shot strategy, which will be the ground truth for the current epoch.
Claim 10’s “train the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 11
Claim 11’s “determine a sensitivity metric value for each of weights in the first updated machine learning model;” is anticipated by Guo, et al., page 4, right column, second full column, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml. The sensitivity of weight matrix Wl, denoted as the influence matrix Infl , can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Claim 11’s “for each subset of weights in a layer of the first updated machine learning model, modify a second predetermined number or percentage of sensitivity metric values;” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 11’s “across the plurality of layers of the first updated machine learning model, select a third predetermined number or percentage the weights of the first updated machine learning model as second weights for pruning by comparing the sensitivity metric values of the weights of the first updated machine learning model, weights of the first updated machine learning model corresponding to the modified sensitivity metric values less likely to be selected for pruning; and” is anticipated by Guo, et al., Abstract, where it recites:
As neural networks get deeper for better performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose eliminating less sensitive filters to compress models. The previous method evaluates neuron importance using the connection matrix gradient in a single shot. To mitigate the sampling bias, we integrate this measure into the previously proposed “pruning while fine-tuning” framework. Besides classification errors, we introduce the difference between the learned and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Our Sensitivity Pruner (SP) adapts the unstructured pruning saliency metric to structured pruning tasks and enables the strategy to be derived sequentially to accommodate the updating sparsity. Experimental results demonstrate that SP significantly reduces the computational cost and the pruned models give comparable or better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets.
Claim 11’s “train the first updated machine learning model with the second weights pruned to generate a second updated machine learning model with a second sparsity of weights higher than the first sparsity of weights.” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 14
Claim 14’s “14. The non-transitory storage medium of claim 11, wherein the instructions to train the machine learning model with the selected weights use second training data that is part of the first training data, and wherein the instructions to train the first updated machine learning model uses third training data that is part of the first training data.” is anticipated by Guo, et al., page 5, right column, next to last full paragraph, where it recites:
3.7. Algorithm summarization
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Claim 17
Claim 17’s “17. The non-transitory storage medium of claim 10, wherein the sensitivity metric value is based on at least one of a magnitude of each of the weights and a gradient associated with each of the weights.” is anticipated by Guo, et al., page 2, left column, first bullet point, where it recites:
We integrate the sensitivity measure from SNIP into the “training while fine-tuning” framework to form a more powerful pruning strategy by adapting the unstructured pruning measure from SNIP to allow filter-level compression. In practice, the sensitivity score can be easily computed as the gradient of the connection mask applied to the weight matrix. Independent of the model structure, the sensitivity score can be applied to most neural networks for pruning purposes.
Claim 18
Claim 18’s “18. The non-transitory storage medium of claim 10, further storing instructions that cause the processor to deploy the first updated machine learning model to perform prediction, inference or creation, wherein the first updated machine learning model is faster than the machine learning model.” is anticipated by Guo, et al., page 6, left column, last full paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradient, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block. To ensure such addition, the last layer of blocks with residual connection ahead is not pruned. In MobileNetV2, point-wise convolution is implemented using the “group” parameter in the PyTorch framework.
Claim 19
Claim 19’s “(a) receiving weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 19’s “(b) determining a sensitivity metric value for each of the weights in the current machine learning model, the sensitivity metric value indicating influence of each of the weights on an output of the current machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 19’s “(c) sparsifying the weights of the machine learning model by selectively zeroing the weights with lowest sensitivity metric values to generate an intermediate machine learning model” is anticipated by Guo, et al., page 4, left column, last partial paragraph, where it recites:
3.3. Sensitivity definition
If we were to prune a connection in weight W l at input channel P , output channel Q, and kernel map position (M, N) , we can simply set the same position in connection matrix to zero, shown as:
Claim 19’s “(d) training the intermediate machine learning model using second training data to generate an updated machine learning model” is anticipated by Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 19’s “(e) determining if the updated machine learning model satisfies a termination condition” Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 19’s “(f) responsive to determining that the termination condition is satisfied, setting the updated machine learning model as the sparse machine learning model; and” is anticipated by Guo, et al., page 6, left column, last partial paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradi- ent, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block.
The model is “set/selected” because it is tested in the prior art.
Claim 19’s “(g) responsive to determining that the termination condition is not satisfied, setting the updated machine learning model as the current machine learning model and repeating (a) through (g).” is anticipated by Guo, et al., page 5, right column, third full paragraph, where it recites:
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Note that in each iteration the model parameters are updated.
Claim 21
Claim 21’s “(a) receiving weights (i.e., pretrained weights) of a plurality of layers of a machine learning model trained using first training data” is anticipated by Guo, et al., page 8, left column, first full paragraph, where it recites:
4.6. Comparisons with SNIP
SNIP is an unstructured pruning algorithm, which requires a mini-batch to generate the strategy and fine-tuning to restore the model performance. In the current hardware setting, only “pseudo” pruning is enabled, and thus, the FLOPs Drop for SNIP is not available. Therefore, we display the performance in a separate table with the Parameter Drop Rate as the substitute for the FLOPs Drop rate. In the original paper, the SNIP is applied to a model with randomized weight initiation. For a fair comparison, we use pre-trained models in SNIP strategy derivation.
Claim 21’s “(b) determining a sensitivity metric value for each of the weights in the machine learning model” is anticipated by Guo, et al., page 4, right column, last full paragraph, where it recites:
With the defined sensitivity measure, we can evaluate the neuron importance by taking the derivative with respect to the connection mask Ml . The sensitivity of weight matrix Wl, denoted as the influence matrix Infl, can be obtained easily in one feed-forward and back-propagate iteration using auto-differentiation provided by common frameworks, such as PyTorch and TensorFlow.
Further, it is anticipated by Guo, et al., page 6, left column, next to last full paragraph, where it recites:
Upon pre-trained models, we insert our sensitivity measuring layer on one model to gather the channel importance and generate a pruning strategy. Then, we apply this strategy to another model to complete “pseudo” pruning, which is trained with the aforementioned two-part loss.
Claim 21’s “(c) sparsifying the weights of the machine learning model by selectively zeroing the weights with lowest sensitivity metric values to generate an intermediate machine learning model” is anticipated by Guo, et al., page 4, left column, last partial paragraph, where it recites:
3.3. Sensitivity definition
If we were to prune a connection in weight W l at input channel P , output channel Q, and kernel map position (M, N) , we can simply set the same position in connection matrix to zero, shown as:
Claim 21’s “(d) training the intermediate machine learning model using second training data to generate an updated machine learning model” is anticipated by Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 21’s “(e) determining if the updated machine learning model satisfies a termination condition” Guo, et al., page 5, left column, third full paragraph, where it recites:
During training, the pruning is implemented by applying Q l , not B l , to the weight matrix using the Hadamard product. The “pseudo”strategy allows the network to revive a neuron even if deemed less sensitive, so the strategy will not stop optimizing until convergence. Moreover, using “ReLU”as the activation function, a connection marked as “pruned”will be projected to 0 while the ”active”connection will maintain its value. Thus, if we remove the pruned channels after training, the prediction will not be affected.
Claim 21’s “(f) responsive to determining that the termination condition is satisfied, setting the updated machine learning model as the sparse machine learning model; and” is anticipated by Guo, et al., page 6, left column, last partial paragraph, where it recites:
We test our method on the three most popular network structures: VGG-16, ResNet-56, and MobileNetV2. In ResNet-56, the residual connection is added to tackle the vanishing gradi- ent, which is implemented using the simple addition of an in- put feature map and output feature map for a certain block.
The model is “set/selected” because it is tested in the prior art.
Claim 21’s “(g) responsive to determining that the termination condition is not satisfied, setting the updated machine learning model as the current machine learning model and repeating (a) through (g).” is anticipated by Guo, et al., page 5, right column, third full paragraph, where it recites:
The SP is recapitulated in the pseudocode 1 . For each training step, the influence matrix is retrieved from one network, and the learned strategy and target strategy are both derived from this matrix, which will later be fed to the loss function with the prediction to encourage pruning for as many filters as intended. Two types of binarization are involved, the first type is the scaled sigmoid with β updated in each iteration and the second type is to discretize the learned strategy after it has been deemed final.
Note that in each iteration the model parameters are updated.
Response to Arguments
Applicant's arguments filed 13 JUL 2026 have been fully considered but they are not persuasive. Specifically, Applicant argues:
Argument 1
Amended claim 1 is directed to incrementally sparsifying (i.e., mathematical steps) a trained machine learning model to generate a sparse machine learning model that executes faster on a processor (see, e.g., the specification, ||||||0067] and [0085]). That is, amended claim 1 is merely based on or involves a mathematical concept for the purpose of generating a sparse machine learning model that runs faster on hardware, and the claim does not recite the mathematical concept itself. Further, amended claim 1 recites the feature of "training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights" that cannot be reduced to a mathematical equation or concept. Hence, amended claim 1 does not recite a mathematical concept for this additional reason.
In AI, the term “sparsifying” relates to the mathematical parameters in a mathematical linear algebra matrix called a “weight matrix” (Applicant discusses these weights later in this argument.)
[0038] Embodiments are related to incrementally increasing sparsity of a machine learning model and training the sparsified machine learning model. The initial machine learning model may be trained as a dense model that includes a large number of active values in its weight tensors. Multiple iterations of sparsifying weights in the weight tensors followed by training of the sparsified machine learning model may be performed to gradually increase the sparsity of the weight tensor while recovering or maintaining the accuracy of the output from the machine learning model. In this way, a sparsifed machine learning model with sparsified weight tensors with an increased speed using reduced computing resources while maintaining the accuracy of the result may be obtained.
Mathematically a “tensor” is a generalized “matrix.” In this case, a mathematical “2-dimensional matrix” is a mathematical “rank-2 tensor”. A “3-dimensional matrix” is a “rank-3 tensor, and so on”. In paragraph [0038], above, Applicant discusses “iterations of sparsifying (i.e., reducing the number of) weights (i.e., parameters) in the weight tensors (i.e., mathematical weight matrices)”.
In paragraph [0042], below, Applicant discusses mathematical “machine learning operations,” such as: “tensor multiplications” (i.e., matrix multiplications), “convolutions” (i.e., shifted matrix multiplications), “tensor dot products” (i.e., matrix multiplications called “dot products”), “etc.” that paragraph is reprinted, below:
[0042] AI accelerator 104 may be a processor that is efficient at performing certain machine learning operations such as tensor multiplications, convolutions, tensor dot products, etc. In various embodiments, accelerator 104 may have different hardware architectures. For example, in one embodiment, accelerator 104 may take the form of field-programmable gate arrays (FPGAs). In another embodiment, accelerator 104 may take the form of application- specific integrated circuits (ASICs), which may include circuits along or circuits in combination with firmware. In some embodiments, a computing device 100 may not have an accelerator 104. Instead, the computing device 100 relies on the CPU 102 or the GPU 106 to run machine learning models.
Sparsification merely reduces the number of mathematical parameters needed to calculate a result.
Applicants argument is unpersuasive.
The rejections stand.
Argument 2
As for mental processes, the claimed invention is not directed to mental processes because the claim contains limitations that cannot be practically performed in the human mind and because the claim improves technology (the “technology’ that Applicant asserts is improved is unspecified. Only the general idea of “technology” is addressed by Applicant) through a practical application of an algorithm. Amended claim 1 recites the features of "modifying a first predetermined number or percentage of sensitivity metric values by adjusting the first predetermined number or percentage of the sensitivity metric values by a predetermined value," "across the plurality of layers of the machine learning model, selecting a second predetermined number or percentage of the weights as first weights for pruning by comparing the sensitivity metric values of the weights," and "training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights," which are performed across the weights of a plurality of layers of a machine learning model and cannot be practically performed in the human mind.
There are numerous mental processes cited in the argument (in conclusory fashion.) For Instance, applicant cites: “…adjusting the first predetermined number or percentage of the sensitivity metric values by a predetermined value…” Simply “adjusting” (e.g., by adding or subtracting) one number by another number is a mental step that is performed in elementary schools, where it is taught, every day.
The argued “selecting a second predetermined number or percentage of the weights as first weights for pruning by comparing the sensitivity metric values of the weights” is also a mental step. Simply “comparing the sensitivity metric values,” and based on that comparison, “selecting a second predetermined number or percentage of the weights as first weights for pruning” is a simple two step mental process.
Applicants argument is unpersuasive.
The rejections stand.
Argument 3
Moreover, amended claim 1 involves a practical application of an algorithm that improves the execution speed of a machine learning model on a processor, which cannot be performed mentally by a human.
Therefore, the claim is not directed to an abstract idea of mathematical concepts or mental processes, and the rejection should be withdrawn.
The argued practical application does not improve the processor itself at all. It only reduces the number of parameters that the processor needs to use to make calculations. A generic processor is used to process purely mathematical operations.
Applicants argument is unpersuasive.
The rejections stand.
Argument 4
Further, under step 2A, Prong 2 of the Alice/Mayo test, the claim as a whole integrates any alleged judicial exception into a practical application. If the claim recites an additional element or combination of elements that integrate the alleged exception into a practical application, then the claim should be considered patent eligible. Amended claim 1 is directed to the practical application of incrementally sparsifying a trained machine learning model in the field of executing machine learning models on processors, which results in a sparse machine learning model that executes faster with reduced computational resources (see, e.g., the specification, T[0067]). Specifically, amended claim 1 includes a combination of elements that relates to an improvement in the functioning of a computer and an improvement to another technology or technical field (see MPEP § 2106.04(d) 1.). In particular, by modifying a subset of the sensitivity metric values by adjusting them by a predetermined value so that the corresponding weights are less likely to be selected for pruning, amended claim 1 generates a sparse machine learning model whose weight layout is partitioned or structured in a manner suitable for accelerated processing by hardware, permitting the processor to skip operations and reducing execution time, latency, and power (see, e.g., the specification, TT[0067], [0085], and [0093]). Accordingly, amended claim 1 is patent eligible under step 2A, Prong 2 of the Alice/Mayo test.
The entire combination is a generic processor that is used to process purely mathematical operations.
Applicants argument is unpersuasive.
The rejections stand.
Argument 5
Assuming, arguendo, that the claims were directed to the alleged abstract idea, the claims recite substantially or significantly more than merely the alleged abstract idea under step 2B of the Alice/Mayo test.
The claim includes additional elements that amount to significantly more in combination with other elements of the claim even when the additional elements themselves do not amount to significantly more (Rapid Litig. Mgmt. V. CellzDirect, 827 F.3d 1042, 1051, 119 USPQ2d 1370, 1375 (Fed. Cir. 2016)). Specifically, the claim recites "modifying a first predetermined number or percentage of sensitivity metric values by adjusting the first predetermined number or percentage of the sensitivity metric values by a predetermined value," "selecting a second predetermined number or percentage of the weights as first weights for pruning by comparing the sensitivity metric values of the weights, weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights," and "training the machine learning model with the first weights pruned to generate a first updated machine learning model with a first sparsity of weights." Individually, each of these additional elements might not amount to significantly more. However, these limitations, when combined, advantageously generate a sparse machine learning model having a partitioned or structured weight layout that executes faster on a processor with reduced execution time, latency, and power (see, e.g., the specification, TT[0067] and [0085]).
The entire combination is a generic processor is used to process purely mathematical operations.
Applicants argument is unpersuasive.
The rejections stand.
Argument 6
Non-conventional and non-generic arrangement of elements that are individually well- known and conventional may amount to substantially more (BASCOM Global Internet Servs. V. AT&T Mobility LLC, 827 F.3d 1341, 1350, 119 USPQ2d 1236, 1242 (Fed. Cir. 2016)). Specifically, the arrangement of adjusting a subset of the sensitivity metric values by a predetermined value before selecting weights for pruning by comparing the sensitivity metric values is unique in that the adjusted sensitivity metric values render the corresponding weights less likely to be selected for pruning, thereby controlling which weights are retained across the plurality of layers. Such a unique arrangement advantageously enables generation of a sparse machine learning model having a weight layout suitable for accelerated processing by hardware (see the specification, TT[0093] and [0095]).
The additional elements of claim 1 constitute non-conventional and non-routine activity as the Office Action has not established otherwise. Specifically, the additional elements in claim 1 include at least "modifying a first predetermined number or percentage of sensitivity metric values by adjusting the first predetermined number or percentage of the sensitivity metric values by a predetermined value" and "weights corresponding to the modified sensitivity metric values less likely to be selected as the first weights." These are additional meaningful limitations that beneficially generate a sparse machine learning model having a weight layout suitable for accelerated processing by hardware. Hence, these added limitations amount to "significantly more" and render the claim eligible for patent protection under 35 U.S.C. § 101.
There is no specified technology that is improved. The claimed invention is a generic processor that is used to process purely mathematical operations.
Applicants argument is unpersuasive.
The rejections stand.
Argument 7
The examples of subject matter eligibility published by the USPTO also support the conclusion that the claimed invention is directed to eligible subject matter. The example set on subject matter eligibility issued by the USPTO on January 7, 2019 includes Example 39 that is similar to the claimed invention in terms of subject matter eligibility. Both Example 39 and the claimed invention relate to processing a machine learning model, such as training a neural network. It is indicated in the examples that the claim of Example 39 is directed to eligible subject matter because the claim, when viewed as a whole, does not recite any judicial exception. The same applies to amended claim 1 of the present invention, which is directed to the specific manner of incrementally sparsifying and training a machine learning model to generate a faster sparse machine learning model. Hence, amended claim 1 should also be patent eligible according to Example 39 of the examples issued by the USPTO on January 7, 2019.
Based on the above, claim 1 and its dependent claims are directed to patent eligible subject matter under 35 U.S.C. § 101.
Applicant’s argued “Sparsification” merely reduces the number of mathematical parameters needed to calculate a mathematical result and to use that result to simply replace the weight parameters of a mathematical weight matrix.
Applicants argument is unpersuasive.
The rejections stand.
Argument 8
Independent claims 10, 19, and 21, as amended, recite similar features as claim 1, as amended. Therefore, claims 10, 19, and 21 and their dependent claims are also directed to patent eligible subject matter under 35 U.S.C. § 101.
Similar arguments for similar claims are similarly unpersuasive.
The rejections stand.
Argument 9
Per claim 1, after a sensitivity metric value is determined for each of the weights, a subset of the determined sensitivity metric values is adjusted by a predetermined value, and the weights are then selected for pruning by comparing the sensitivity metric values, such that the weights corresponding to the adjusted sensitivity metric values are less likely to be selected for pruning.
This feature is advantageous, among other reasons, because adjusting the sensitivity metric values of a subset of the weights by a predetermined value decreases the likelihood that those weights are pruned, thereby steering the resulting sparse weight layout toward a partitioned or structured pattern suitable for accelerated processing by hardware and enabling the resulting model to execute faster on a processor with reduced execution time, latency, and power (see, e.g., the specification, |||||0067], [0085], [0093], and [0095]).
Guo fails to disclose this feature. Guo relates to a filter-level compression algorithm for deep neural networks that measures the sensitivity of filters and prunes filters having low sensitivity (see Guo, abstract). Guo at best discloses computing a sensitivity measure by taking a derivative with respect to a connection mask and pruning the weights having the lowest sensitivity metric values (see Guo, page 4, right column). Guo determines a sensitivity measure and directly prunes on that basis. That is, Guo does not disclose the separate, subsequent step recited in amended claim 1 of adjusting a subset of the already-determined sensitivity metric values by a predetermined value so that the weights corresponding to the adjusted sensitivity metric values are less likely to be selected for pruning. This distinction is corroborated by the Office Action itself, which indicates claims 6, 15, and former claim 20 as containing subject matter patentable over the prior art on the ground that Guo "fails to expressly teach" increasing determined sensitivity metric values of selected weights by a predetermined value-confirming that the disclosure of Guo stops at computing sensitivity and pruning the lowest, and does not extend to adjusting determined sensitivity metric values to protect selected weights from pruning.
Hence, amended claim 1 and its dependent claims are patentably distinguishable over Guo.
Independent claims 10, 19, and 21, as amended, recite similar features as claim 1, as amended. In particular, amended claim 10 recites "modify a first predetermined number or percentage of sensitivity metric values by adjusting the first predetermined number or percentage of the sensitivity metric values by a predetermined value," and each of amended claims 19 and 21 recites "adjusting sensitivity metric values of the selected weights." Therefore, claims 10, 19, and 21 and their dependent claims are also patentably distinguishable over Guo for the same reasons.
Accordingly, withdrawal of the rejection is respectfully requested.
The 35 U.S.C § 102 rejections are withdrawn.
Argument 10
Allowability of Claims Is Acknowledged
Allowability of claims 3-4, 6-7, 12-13, and 15-16 is respectfully acknowledged.
The 35 U.S.C § 102 rejections are withdrawn, however, the 35 U.S.C § 101 rejections stand.
Argument 11
New Claims Are Added
New claim 22 is added herein. Examination of new claim 22 is respectfully requested.
The 35 U.S.C § 102 rejections are withdrawn, however, this claim is rejected under 35 U.S.C § 101.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiries concerning this communication or earlier communications from the examiner should be directed to Wilbert L. Starks, Jr., who may be reached Monday through Friday, between 8:00 a.m. and 5:00 p.m. EST. or via telephone at (571) 272-3691 or email: Wilbert.Starks@uspto.gov.
If you need to send an Official facsimile transmission, please send it to (571) 273-8300.
If attempts to reach the examiner are unsuccessful the Examiner’s Supervisor (SPE), Kakali Chaki, may be reached at (571) 272-3719.
Hand-delivered responses should be delivered to the Receptionist @ (Customer Service Window Randolph Building 401 Dulany Street, Alexandria, VA 22313), located on the first floor of the south side of the Randolph Building.
Finally, information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Moreover, status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have any questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) toll-free @ 1-866-217-9197.
/WILBERT L STARKS/
Primary Examiner, Art Unit 2122
WLS
27 SEP 2026
