DETAILED ACTION
This action is in response to the application filed 10/13/2023. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 16 and 19 are objected to because of the following informalities:
In claim 16, “sparsifying the original set weights” should read “sparsifying the original set of weights”.
In claim 19, “by setting a prescribed number of number of weights” should read “by setting a prescribed number of weights”.
Appropriate correction is required.
Claim Interpretation
In [0086] of the instant specification, the applicant notes, “However, the specific term ‘computer-readable storage medium’ or ‘storage device’ expressly excludes propagated signals per se in transit, while including all other forms of computer-readable media; a computer-readable storage medium or storage device is ‘non- transitory’ in this regard.” Therefore, the examiner notes that the computer readable storage medium is understood to mean non-transitory. The examiner further respectfully recommends amending the claim language to indicate the computer-readable storage medium is non-transitory to further aid in the understanding of the reading public.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7, 17, and 18-20 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 7, “all but one of which will resolve to zero” in the last line of claim 7 is indefinite. Specifically, it is unclear as to whether the mask value, the weights, or the input embedding resolve to zero. Further, it is unclear as how all but one of the mask values and input embeddings resolve to zero when claim 1 and claim 7 only recite one mask value and one input embedding. Additionally, it is unclear as to how all but one of the weights resolve to zero when claim 1 recites multiple selected and non-selected weights in which the non-selected weights represent weights that have been set to zero. For purposes of examination, Examiner has interpreted this limitation to mean some of the weights resolve to zero.
Regarding claim 17, “all but one of which will resolve to zero” in the last line of claim 7 is indefinite. Specifically, it is unclear as to whether the mask value, the weights, or the input embedding resolve to zero. Further, it is unclear as how all but one of the mask values and input embeddings resolve to zero when claim 1 and claim 7 only recite one mask value and one input embedding. Additionally, it is unclear as to how all but one of the weights resolve to zero when claim 1 recites multiple selected and non-selected weights in which the non-selected weights represent weights that have been set to zero. For purposes of examination, Examiner has interpreted this limitation to mean some of the weights resolve to zero.
Regarding claim 18, “all but one of which will resolve to zero” in the last line of claim 7 is indefinite. Specifically, it is unclear as to whether the mask value, the weights, or the input embedding resolve to zero. Further, it is unclear as how all but one of the mask values and input embeddings resolve to zero when claim 1 and claim 7 only recite one mask value and one input embedding. Additionally, it is unclear as to how all but one of the weights resolve to zero when claim 1 recites multiple selected and non-selected weights in which the non-selected weights represent weights that have been set to zero. For purposes of examination, Examiner has interpreted this limitation to mean some of the weights resolve to zero.
Claims 19-20 are rejected for at least the same reason as claim 18 since claims 19-20 depend on claim 18.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites a method and is thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 1 recites
the selected set of weights and the mask having been produced by a sparsification process that operates on an original set of weights, the sparsification process discriminating between the selected set of weights and a non-selected set of weights, (This limitation is a mental process as it encompasses a human mentally discriminating between the selected set of weights and a non-selected set of weights and is thus a judgement.)
transforming the input embedding by performing computations directly on the selected set of weights and the mask, to produce an output result (This limitation is a mental process as it encompasses a human mentally transforming the input embedding and is thus an evaluation.)
Therefore, claim 1 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 1 further recites additional elements of
for executing a machine-trained model (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
receiving a selected set of weights and a mask (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
storing the selected set of weights and the mask in memory; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
receiving an input embedding; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
in a processor (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 1 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 1 do not provide significantly more than the abstract idea itself, taken alone and in combination because
for executing a machine-trained model uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
receiving a selected set of weights and a mask is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
storing the selected set of weights and the mask in memory; is the well understood, routine, and conventional activity of “storing and retrieving information in memory” (see MPEP 2106.05(d)(II); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
receiving an input embedding; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
in a processor uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 1 is subject-matter ineligible.
Regarding Claim 2:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 2 recites
wherein the non-selected set of weights are weights that represent zero values. (This limitation is a mental process as it further defines the mental process of discriminating between the selected set of weights and a non-selected set of weights from claim 1.)
Therefore, claim 2 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 2 does not further recite any additional elements. Therefore, claim 2 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 2 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 2 is subject-matter ineligible.
Regarding Claim 3:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 3 recites
wherein the sparsification process identifies the combined set of weights by setting a prescribed number of original weights in the original set of weights to the non-selected set of weights, based on a prescribed pattern (This limitation is a mental process as it encompasses a human mentally identifies the combined set of weights by setting a prescribed number and is thus an observation.)
Therefore, claim 3 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 3 does not further recite any additional elements. Therefore, claim 3 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 3 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 3 is subject-matter ineligible.
Regarding Claim 4:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 4 recites
wherein the prescribed pattern specifies, for a particular group of weights, a number of selected weights to be included in the particular group, (This limitation is a mental process as it further defines the prescribed pattern from claim 3 that is used to identify the combined set of weights.)
wherein a particular mask value in the mask specifies a location of a particular selected weight in the particular group of weights (This limitation is a mental process as it encompasses a human mentally specifying a location using a value and is thus an evaluation.)
Therefore, claim 4 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 4 does not further recite any additional elements. Therefore, claim 4 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 4 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 4 is subject-matter ineligible.
Regarding Claim 5:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 5 recites
wherein a particular mask value in the mask identifies whether a particular pair of weights in the combined set of weights includes a selected weight as a first member or a second member of the particular pair (This limitation is a mental process as it encompasses a human mentally identifying whether a pair of weights includes a selected weight using a value and is thus an observation.)
Therefore, claim 5 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 5 does not further recite any additional elements. Therefore, claim 6 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 5 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 5 is subject-matter ineligible.
Regarding Claim 6:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 6 recites the same abstract ideas as claim 1. Therefore, claim 6 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 6 further recites additional elements of
wherein the mask has plural mask values, each mask value being represented by a single bit in the mask (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 6 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 6 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the mask has plural mask values, each mask value being represented by a single bit in the mask specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 6 is subject-matter ineligible.
Regarding Claim 7:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 7 recites
wherein the transforming uses a particular mask value of the mask to evaluate plural ways of applying a particular weight of the selected set of weights to the input embedding, all but one of which will resolve to zero (This limitation is a mental process as it encompasses a human mentally evaluating plural ways of applying a weight to the input embedding and is thus an evaluation.)
Therefore, claim 7 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 7 does not further recite any additional elements. Therefore, claim 7 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 7 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 7 is subject-matter ineligible.
Regarding Claim 8:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 8 recites
multiplying a particular mask value by a particular weight of the selected set of weights and a particular element of the input embedding, to produce a first intermediate value; (This limitation is a mental/mathematical process as it encompasses a human mentally multiplying a value by a weight..)
multiplying a binary opposite of the particular mask value by the particular weight and another particular element of the input embedding, to produce a second intermediate value; (This limitation is a mental/mathematical process as it encompasses a human mentally multiplying a binary opposite by a weight.)
adding the first intermediate value to the second intermediate value (This limitation is a mental/mathematical process as it encompasses a human mentally adding intermediate values together.)
Therefore, claim 8 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 8 does not further recite any additional elements. Therefore, claim 8 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 8 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 8 is subject-matter ineligible.
Regarding Claim 9:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 9 recites the same abstract ideas as claim 1. Therefore, claim 9 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 9 further recites additional elements of
wherein the mask has mask values that are separate from the selected set of weights. (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 9 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 9 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the mask has mask values that are separate from the selected set of weights. specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 9 is subject-matter ineligible.
Regarding Claim 10:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 10 recites the same abstract ideas as claim 1. Therefore, claim 10 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 10 further recites additional elements of
wherein the mask has mask values that are incorporated into weights in the selected set of weights. (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 10 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 10 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the mask has mask values that are incorporated into weights in the selected set of weights. specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 10 is subject-matter ineligible.
Regarding Claim 11:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 11 recites
wherein the method incorporates a particular mask value into a particular weight of the selected set of weights by adding an offset to the particular weight that reflects the particular mask value. (This limitation is a mental process as it encompasses a human mentally incorporating a mask value into a weight by adding an offset and is thus an evaluation.)
Therefore, claim 11 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 11 does not further recite any additional elements. Therefore, claim 11 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 11 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 11 is subject-matter ineligible.
Regarding Claim 12:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 12 recites
wherein the transforming computationally extracts the particular mask value from the particular weight. (This limitation is a mental process as it encompasses a human mentally computationally extracting the mask value and is thus an evaluation.)
Therefore, claim 12 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 12 does not further recite any additional elements. Therefore, claim 12 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 12 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 12 is subject-matter ineligible.
Regarding Claim 13:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 13 recites the same abstract ideas as claim 1. Therefore, claim 13 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 13 further recites additional elements of
wherein the machine-trained model has plural layers, and wherein the method is performed for transformations executed by each layer of the plural layers. (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 13 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 13 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the machine-trained model has plural layers, and wherein the method is performed for transformations executed by each layer of the plural layers. uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 13 is subject-matter ineligible.
Regarding Claim 14:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 14 recites the same abstract ideas as claim 1. Therefore, claim 14 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 14 further recites additional elements of
wherein the machine-trained model is a transformer-based model. (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 14 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 14 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the machine-trained model is a transformer-based model. uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 14 is subject-matter ineligible.
Regarding Claim 15:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 15 recites the same abstract ideas as claim 1. Therefore, claim 15 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 15 further recites additional elements of
wherein the processor is a graphics processing unit or a neural processing unit. (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 15 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 15 do not provide significantly more than the abstract idea itself, taken alone and in combination because
wherein the processor is a graphics processing unit or a neural processing unit. uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 15 is subject-matter ineligible.
Regarding Claim 16:
Subject Matter Eligibility Analysis Step 1:
Claim 16 recites a computer-readable storage medium and is thus a product of manufacture, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 16 recites
sparsifying the original set weights in sparsification process, to produce a selected set of weights and a mask (This limitation is a mental process as it encompasses a human mentally sparsifying a set of weights and is thus an evaluation.)
the sparsification process discriminating between the selected set of weights and a non-selected set of weights, (This limitation is a mental process as it encompasses a human mentally discriminating between the selected set of weights and a non-selected set of weights and is thus a judgement.)
transforming the input embedding by performing computations directly on the selected set of weights and the mask, to produce an output result (This limitation is a mental process as it encompasses a human mentally transforming the input embedding and is thus an evaluation.)
the transforming computationally duplicating an effect of operating on the combined set of weights without a process of reconstituting the non-selected set of weights … prior to the transforming (This limitation is a mental process as it encompasses a human mentally transforming the input embedding without reconstituting the set of weights.).
Therefore, claim 16 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 16 further recites additional elements of
A computer-readable storage medium for storing computer-readable instructions, a processing system executing the computer-readable instructions to perform operations, the operations comprising: (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
receiving an original set of weights (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
storing the selected set of weights and the mask in memory; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
receiving an input embedding; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
in memory (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 16 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 16 do not provide significantly more than the abstract idea itself, taken alone and in combination because
A computer-readable storage medium for storing computer-readable instructions, a processing system executing the computer-readable instructions to perform operations, the operations comprising uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
receiving an original set of weights is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
storing the selected set of weights and the mask in memory; is the well understood, routine, and conventional activity of “storing and retrieving information in memory” (see MPEP 2106.05(d)(II); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
receiving an input embedding; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
in memory uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 16 is subject-matter ineligible.
Regarding claim 17, claim 17 recites substantially similar limitations to claim 7, and is therefore rejected under the same analysis.
Regarding Claim 18:
Subject Matter Eligibility Analysis Step 1:
Claim 18 recites a computing system and is thus an apparatus, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 18 recites
determining whether a target item is supported by a source item (This limitation is a mental process as it encompasses a human mentally determining whether a target item is supported by a source item and is thus a judgement.)
the selected set of weights and the mask having been produced by a sparsification process that operates on an original set of weights, the sparsification process discriminating between the selected set of weights and a non-selected set of weights, (This limitation is a mental process as it encompasses a human mentally discriminating between the selected set of weights and a non-selected set of weights and is thus a judgement.)
transforming the input embedding by performing computations directly on the selected set of weights and the mask, to produce an output result (This limitation is a mental process as it encompasses a human mentally transforming the input embedding and is thus an evaluation.)
the transforming uses a particular mask value of the mask to evaluate plural ways of applying a particular weight of the selected set of weights to the input embedding, all but one of which will resolve to zero (This limitation is a mental process as it encompasses a human mentally evaluating plural ways of applying a weight to the input embedding and is thus an evaluation.)
Therefore, claim 18 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 18 further recites additional elements of
A computing system for determining whether a target item is supported by a source item, comprising: a memory; a processing system for executing computer-readable instructions, to perform operations including: (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
receiving a selected set of weights and a mask (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
storing the selected set of weights and the mask in memory; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
receiving an input embedding; (This element does not integrate the abstract idea into a practical application because it recites insignificant extra-solution activity of data gathering (see MPEP 2106.05(g)).)
Therefore, claim 18 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 18 do not provide significantly more than the abstract idea itself, taken alone and in combination because
A computing system for determining whether a target item is supported by a source item, comprising: a memory; a processing system for executing computer-readable instructions, to perform operations including uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
receiving a selected set of weights and a mask is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights specifies a particular technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
storing the selected set of weights and the mask in memory; is the well understood, routine, and conventional activity of “storing and retrieving information in memory” (see MPEP 2106.05(d)(II); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;
receiving an input embedding; is the well understood, routine, and conventional activity of “transmitting or receiving data over a network” (see MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
Therefore, claim 18 is subject-matter ineligible.
Regarding Claim 19:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 19 recites
wherein the sparsification process identifies the combined set of weights by setting a prescribed number of original weights in the original set of weights to the non-selected set of weights, based on a prescribed pattern (This limitation is a mental process as it encompasses a human mentally identifies the combined set of weights by setting a prescribed number and is thus an observation.)
wherein the prescribed pattern specifies, for a particular group of weights, a number of selected weights to be included in the particular group, (This limitation is a mental process as it further defines the prescribed pattern from claim 3 that is used to identify the combined set of weights.)
wherein a particular mask value in the mask specifies a location of a particular selected weight in the particular group of weights (This limitation is a mental process as it encompasses a human mentally specifying a location using a value and is thus an evaluation.)
Therefore, claim 19 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 19 does not further recite any additional elements. Therefore, claim 19 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 19 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 19 is subject-matter ineligible.
Regarding Claim 20:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 20 recites
wherein the prescribed number is 50 percent (This limitation is a mental process as it further defines the mental process of setting a prescribed number of claim 19.)
Therefore, claim 20 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 20 does not further recite any additional elements. Therefore, claim 20 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 20 does not provide significantly more than the abstract idea itself, taken alone and in combination. Therefore, claim 20 is subject-matter ineligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Frantar et al. (“SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot”) (hereafter referred to as Frantar).
Regarding claim 1, Frantar teaches
A method for executing a machine-trained model, comprising: receiving a selected set of weights and a mask (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask.),
the selected set of weights and the mask having been produced by a sparsification process that operates on an original set of weights, the sparsification process discriminating between the selected set of weights and a non-selected set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask. Examiner further notes that Algorithm 1 is the sparsification process and the binary pruning mask discriminates between the selected set of weights equal to 1 and the non-selected weights equal to 0.),
and the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask. Examiner further notes that Algorithm 1 is the sparsification process and the binary pruning mask discriminates between the selected set of weights equal to 1 and the non-selected weights equal to 0. Examiner further notes that the i and j of the for loops are the positions.);
storing the selected set of weights and the mask in memory (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory” (Frantar, page 6, 4. Experiments Setup.). Examiner notes that since all experiments are done on a GPU with memory, the weights and mask of Algorithm 1 is stored in memory.);
receiving an input embedding (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that H is the input embedding. Examiner further notes that according to paragraph 0039 of the specification, “the input embedding represents an input submission of any kind”.);
and in a processor, transforming the input embedding by performing computations directly on the selected set of weights and the mask, to produce an output result (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory” (Frantar, page 6, 4. Experiments Setup.). Examiner notes that Algorithm 1 transforms H, the input embedding by performing computations on the weights, W, and the mask, M. Examiner further notes that W in the last line the last line of Algorithm 1 is the output result.).
Regarding claim 2, Frantar teaches
The method of claim 1, wherein the non-selected set of weights are weights that represent zero values(Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner further notes that Algorithm 1 or Sparse GPT is the sparsification process and the binary pruning mask discriminates between the selected set of weights equal to 1 and the non-selected weights equal to 0.).
Regarding claim 3, Frantar teaches
The method of claim 1, wherein the sparsification process identifies the combined set of weights by setting a prescribed number of original weights in the original set of weights to the non-selected set of weights, based on a prescribed pattern (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches” (Frantar, page 1, Abstract). Examiner notes that the sparsification process is SparseGPT. Examiner further notes that p is the prescribed number in the original weights being set to the non-selected weights. Examiner notes that the prescribed pattern is the semi-structured patterns of 2:4 and 4:8).
Regarding claim 4, Frantar teaches,
The method of 3 wherein the prescribed pattern specifies, for a particular group of weights, a number of selected weights to be included in the particular group (Frantar, page 1, Abstract, “We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches” where “SparseGPT is also easily adapted to semi-structured patterns such as the popular n:m sparsity format (Zhou et al., 2021; Hubara et al., 2021a) which delivers speedups in its 2:4 implementation on Ampere NVIDIA GPUs. Specifically, every consecutive m weights should contain exactly n zeros” (Frantar, page 5, 3.3 Extension to Semi-Structured Sparsity). Examiner notes that the prescribed pattern is the semi-structured pattern of 2:4 and 4:8 and the selected weights are the weights that aren’t 0.),
and wherein a particular mask value in the mask specifies a location of a particular selected weight in the particular group of weights (Frantar, page 4, Figure 4,
PNG
media_image2.png
276
762
media_image2.png
Greyscale
Examiner notes that the mask M is the mask and each cell in mask M is a mask value which specifies a location of a selected weight in light gray in W.).
Regarding claim 5, Frantar teaches
The method of claim 1, wherein a particular mask value in the mask identifies whether a particular pair of weights in the combined set of weights includes a selected weight as a first member or a second member of the particular pair (Frantar, page 4, Figure 4,
PNG
media_image2.png
276
762
media_image2.png
Greyscale
Examiner notes that the mask M is the mask and each cell in mask M is a mask value. Examiner further notes that each cell in W is a weight and that a particular mask value identifies the first member or first cell of the first column of a particular pair of weights, which includes the first and second cell of the first column, as a selected weight.).
Regarding claim 6, Frantar teaches
The method of claim 1, wherein the mask has plural mask values, each mask value being represented by a single bit in the mask(Frantar, page 4, Figure 4,
PNG
media_image2.png
276
762
media_image2.png
Greyscale
“throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1).Examiner notes that the mask M is the mask and each cell in mask M is a mask value which is either a 0 at the white cells or a 1 on the gray cells. ).
Regarding claim 7, Frantar teaches
The method of claim 1, wherein the transforming uses a particular mask value of the mask to evaluate plural ways of applying a particular weight of the selected set of weights to the input embedding, all but one of which will resolve to zero (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and Frantar, page 4, Figure 4,
PNG
media_image2.png
276
762
media_image2.png
Greyscale
where “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that M is the mask, the plural ways is the matrix multiplication occurring in the first update line of Algorithm 1 where E includes the mask and weight and is being multiplied by the input embedding H. Examiner further notes that “all but one of which will resolve to 0” as stated in the above 112(b) section is interpreted as some weights resolve to 0. In this case, the last line of algorithm 1 reads on this.).
Regarding claim 8, Frantar teaches
The method of claim 1, wherein the transforming comprises: multiplying a particular mask value by a particular weight of the selected set of weights and a particular element of the input embedding, to produce a first intermediate value (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and Frantar, page 4, Figure 4 (see below),
PNG
media_image2.png
276
762
media_image2.png
Greyscale
Examiner notes that the freeze weights line of Algorithm 1 shows this limitation since E has both the weight and the input embedding in it and is multiplied by M, the mask. Examiner further notes that the first intermediate value is -M:,j • E:,j-i);
multiplying a binary opposite of the particular mask value by the particular weight and another particular element of the input embedding, to produce a second intermediate value (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that the freeze weights line of Algorithm 1 shows this limitation since E has both the weight and the input embedding in it and is multiplied by 1, which is a binary opposite of the mask. Examiner further notes that the first intermediate value is E:,j-i);
and adding the first intermediate value to the second intermediate value (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that the freeze weights line of Algorithm 1 shows this limitation E:,j-i and -M:,j • E:,j-i are added together and assigned to E:,j-i.).
Regarding claim 9, Frantar teaches
The method of claim 1, wherein the mask has mask values that are separate from the selected set of weights (Frantar, page 4, Figure 4,
PNG
media_image2.png
276
762
media_image2.png
Greyscale
“throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1).Examiner notes that the mask M is the mask and each cell in mask M is a mask value which is either a 0 at the white cells or a 1 on the gray cells. Examiner further notes that the mask values are separate from the weights.).
Regarding claim 10, Frantar teaches
The method of claim 1, wherein the mask has mask values that are incorporated into weights in the selected set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that the second update line shows this limitation since W includes E which was calculated using M, the mask values).
Regarding claim 11, Frantar teaches
The method of claim 10, wherein the method incorporates a particular mask value into a particular weight of the selected set of weights by adding an offset to the particular weight that reflects the particular mask value (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that the second update line shows this limitation since W includes E which was calculated using M, the mask values. Examiner notes that the second update line has -E•
H
j
,
j
:
(
i
+
B
)
-
1
which is added as an offset to the weight.).
Regarding claim 12, Frantar teaches
The method of claim 11, wherein the transforming computationally extracts the particular mask value from the particular weight (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that the if statement computationally extracts the particular mask value from the particular weight.).
Regarding claim 13, Frantar teaches
The method of claim 1, wherein the machine-trained model has plural layers, and wherein the method is performed for transformations executed by each layer of the plural layers (Frantar, page 6, 4. Experiments, Setup. “Similar to Yao et al. (2022); Frantar et al. (2022a), we sparsify Transformer layers sequentially in order, which significantly reduces memory requirements.”).
Regarding claim 14, Frantar teaches
The method of claim 1, wherein the machine-trained model is a transformer-based model (Frantar, page 6, 4. Experiments, Setup. “Similar to Yao et al. (2022); Frantar et al. (2022a), we sparsify Transformer layers sequentially in order, which significantly reduces memory requirements.”).
Regarding claim 15, Frantar teaches
The method of claim 1, wherein the processor is a graphics processing unit or a neural processing unit (Frantar, page 6, 4. Experiments Setup. “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory.”).
Regarding claim 16, Frantar teaches
A computer-readable storage medium for storing computer-readable instructions, a processing system executing the computer-readable instructions to perform operations, the operations comprising (Frantar, page 6, 4. Experiments Setup. “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory.”):
receiving an original set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask.),
sparsifying the original set weights in a sparsification process, to produce a selected set of weights and a mask (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask.),
the sparsification process discriminating between the selected set of weights and a non-selected set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask. Examiner further notes that Algorithm 1 is the sparsification process and the binary pruning mask discriminates between the selected set of weights equal to 1 and the non-selected weights equal to 0.),
and the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask. Examiner further notes that Algorithm 1 is the sparsification process and the binary pruning mask discriminates between the selected set of weights equal to 1 and the non-selected weights equal to 0. Examiner further notes that the i and j of the for loops are the positions.);
storing the selected set of weights and the mask in memory (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory” (Frantar, page 6, 4. Experiments Setup.). Examiner notes that since all experiments are done on a GPU with memory, the weights and mask of Algorithm 1 is stored in memory.);
receiving an input embedding (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that H is the input embedding. Examiner further notes that according to paragraph 0039 of the specification, “the input embedding represents an input submission of any kind”.);
transforming the input embedding by performing computations directly on the selected set of weights and the mask, to produce an output result (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory” (Frantar, page 6, 4. Experiments Setup.). Examiner notes that Algorithm 1 transforms H, the input embedding by performing computations on the weights, W, and the mask, M. Examiner further notes that W in the last line the last line of Algorithm 1 is the output result.)
the transforming computationally duplicating an effect of operating on the combined set of weights without a process of reconstituting the non-selected set of weights in memory prior to the transforming (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models” (Frantar, page 1, Abstract). Examiner notes that the retraining is the reconstituting the non-selected set of weights in memory. Examiner further notes that the updating of the model occurs during the transformation according to Algorithm 1 instead of prior. ).
Regarding claim 17, claim 17 recites substantially similar limitations to claim 7, and is therefore rejected under the same analysis.
Regarding claim 18, Frantar teaches
A computing system for determining whether a target item is supported by a source item, comprising: a memory; a processing system for executing computer-readable instructions, to perform operations including (Frantar, page 6, 4. Experiments Setup. “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory.”):
receiving a selected set of weights and a mask (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask.),
the selected set of weights and the mask having been produced by a sparsification process that operates on an original set of weights, the sparsification process discriminating between the selected set of weights and a non-selected set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask. Examiner further notes that Algorithm 1 is the sparsification process and the binary pruning mask discriminates between the selected set of weights equal to 1 and the non-selected weights equal to 0.),
and the mask describing positions of the selected set of weights and the non-selected set of weights among a combined set of weights (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1 (see below),
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that W is the set of weights and M or the adaptive mask selection is the mask. Examiner further notes that Algorithm 1 is the sparsification process and the binary pruning mask discriminates between the selected set of weights equal to 1 and the non-selected weights equal to 0. Examiner further notes that the i and j of the for loops are the positions.);
storing the selected set of weights and the mask in memory (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory” (Frantar, page 6, 4. Experiments Setup.). Examiner notes that since all experiments are done on a GPU with memory, the weights and mask of Algorithm 1 is stored in memory.);
receiving an input embedding (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
Examiner notes that H is the input embedding. Examiner further notes that according to paragraph 0039 of the specification, “the input embedding represents an input submission of any kind”.);
and transforming the input embedding by performing computations directly on the selected set of weights and the mask, to produce an output result (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “All pruning experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory” (Frantar, page 6, 4. Experiments Setup.). Examiner notes that Algorithm 1 transforms H, the input embedding by performing computations on the weights, W, and the mask, M. Examiner further notes that W in the last line the last line of Algorithm 1 is the output result.)
the transforming using a particular mask value of the mask to evaluate plural ways of applying a particular weight of the selected set of weights to the input embedding, all but one of which will resolve to zero (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and Frantar, page 4, Figure 4,
PNG
media_image2.png
276
762
media_image2.png
Greyscale
where “throughout the paper, by sparsity mask for a given tensor we mean a binary tensor of the same dimensions, with 0 at the indices of the sparsified entries, and 1 at the other indices” (Frantar, page 2, Footnote 1). Examiner notes that M is the mask, the plural ways is the matrix multiplication occurring in the first update line of Algorithm 1 where E includes the mask and weight and is being multiplied by the input embedding H. Examiner further notes that “all but one of which will resolve to 0” as stated in the above 112(b) section is interpreted as some weights resolve to 0. In this case, the last line of algorithm 1 reads on this.).
Regarding claim 19, Frantar teaches
The computing system of claim 18, wherein the sparsification process identifies the combined set of weights by setting a prescribed number of original weights in the original set of weights to the non-selected set of weights, based on a prescribed pattern (Frantar, page 5, 3.4 Full Algorithm Pseudocode, Algorithm 1,
PNG
media_image1.png
785
752
media_image1.png
Greyscale
and “We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches” (Frantar, page 1, Abstract). Examiner notes that the sparsification process is SparseGPT. Examiner further notes that p is the prescribed number in the original weights being set to the non-selected weights. Examiner notes that the prescribed pattern is the semi-structured patterns of 2:4 and 4:8)
wherein the prescribed pattern specifies, for a particular group of weights, a number of selected weights to be included in the particular group (Frantar, page 1, Abstract, “We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches” where “SparseGPT is also easily adapted to semi-structured patterns such as the popular n:m sparsity format (Zhou et al., 2021; Hubara et al., 2021a) which delivers speedups in its 2:4 implementation on Ampere NVIDIA GPUs. Specifically, every consecutive m weights should contain exactly n zeros” (Frantar, page 5, 3.3 Extension to Semi-Structured Sparsity). Examiner notes that the prescribed pattern is the semi-structured pattern of 2:4 and 4:8 and the selected weights are the weights that aren’t 0.),
and wherein a particular mask value in the mask specifies a location of a particular selected weight in the particular group of weights (Frantar, page 4, Figure 4,
PNG
media_image2.png
276
762
media_image2.png
Greyscale
Examiner notes that the mask M is the mask and each cell in mask M is a mask value which specifies a location of a selected weight in light gray in W.).
Regarding claim 20, Frantar teaches
The computing system of claim 19, wherein the prescribed number is 50 percent (Frantar, page 1, Abstract, “We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Sanh et al. (“Movement Pruning: Adaptive Sparsity by Fine-Tuning”) also discloses a pruning method to a transformer base machine learning model.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAITLYN R LAU whose telephone number is (571)272-1429. The examiner can normally be reached Monday - Thursday: 7:15 am - 5:15 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.L./Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights