Detailed Action
Claims 1-30 are presently pending. Claims 1-10 and 16-25 are elected. Claims 11-15 and 26-30 are withdrawn from consideration.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) was submitted on 08/30/2024 and 08/06/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Election/Restrictions
Restriction to one of the following inventions is required under 35 U.S.C. 121:
Claims 1-10 and 16-25, drawn to a system for generating, via a first attention layer, a first attention map, identify a first subset of tokens more relevant to a second attention layer and a second subset of tokens less relevant to the second attention layer based on the first attention map, and generate a second attention map based on the first subset of tokens, and generating an inference based on the second attention map and the second subset of tokens, classified in G06N5/04.
Claims 11-15 and 26-30, drawn to a system and a method for generating a plurality of ground-truth attention maps for a set of inputs in a training data set using a machine learning model; training a machine learning model to generate a predicted attention map by minimizing Kullback-Leibler divergence loss; and deploying the trained token prediction model, classified in G06N3/09.
The inventions are independent or distinct, each from the other because:
Inventions I and II are directed to related processes. The related inventions are distinct if: (1) the inventions as claimed are either not capable of use together or can have a materially different design, mode of operation, function, or effect; (2) the inventions do not overlap in scope, i.e., are mutually exclusive; and (3) the inventions as claimed are not obvious variants. See MPEP § 806.05(j). In the instant case, the inventions as claimed
(1) have different modes of operations and functions (Group I recites generating inference by generating first attention map, a token prediction, and second attention map using a first attention layer, a token prediction model, and a second attention layer. Group II recites training a machine learning prediction model by first generating a plurality of ground-truth attention maps for a set of inputs in a training data set using a machine learning model, calculating and minimizing Kullback-Leibler divergence loss and deploying the trained model.)
(2) do not overlap in scope (each invention of groups I and II contain features which do not appear in any of the other inventions) and
(3) the inventions of groups I and II are not obvious variants of one another.
Furthermore, the inventions as claimed do not encompass overlapping subject matter and there is nothing of record to show them to be obvious variants.
Restriction for examination purposes as indicated is proper because all the inventions listed in this action are independent or distinct for the reasons given above and there would be a serious search and/or examination burden if restriction were not required because one or more of the following reasons apply:
Groups I and II have different, non-overlapping series of steps which would require a serious search and examination burden.
Applicant is reminded that upon the cancelation of claims to a non-elected invention, the inventorship must be corrected in compliance with 37 CFR 1.48(a) if one or more of the currently named inventors is no longer an inventor of at least one claim remaining in the application. A request to correct inventorship under 37 CFR 1.48(a) must be accompanied by an application data sheet in accordance with 37 CFR 1.76 that identifies each inventor by his or her legal name and by the processing fee required under 37 CFR 1.17(i).
Applicant is advised that the reply to this requirement to be complete must include (i) an election of an invention to be examined even though the requirement may be traversed (37 CFR 1.143) and (ii) identification of the claims encompassing the elected invention.
The election of an invention may be made with or without traverse. To reserve a right to petition, the election must be made with traverse. If the reply does not distinctly and specifically point out supposed errors in the restriction requirement, the election shall be treated as an election without traverse. Traversal must be presented at the time of election in order to be considered timely. Failure to timely traverse the requirement will result in the loss of right to petition under 37 CFR 1.144. If claims are added after the election, applicant must indicate which of these claims are readable upon the elected invention.
Should applicant traverse on the ground that the inventions are not patentably distinct, applicant should submit evidence or identify such evidence now of record showing the inventions to be obvious variants or clearly admit on the record that this is the case. In either instance, if the examiner finds one of the inventions unpatentable over the prior art, the evidence or admission may be used in a rejection under 35 U.S.C. 103 or pre-AIA 35 U.S.C. 103(a) of the other invention.
During a telephone conversation with Steven Roberts (Registration No. 72,615) on 22 Sept 2026, a provisional election was made without traverse to prosecute the invention of Group I, claims 1-10 and 16-25. Affirmation of this election must be made by applicant in replying to this Office Action. Claims 11-15 and 26-30 are withdrawn from further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7 and 22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 7, the claim recites “wherein the first attention map and the second attention map comprise attention maps for a first attention head in the machine learning model, and wherein the token prediction model comprises a model specific to the first attention head.” It is unclear what constitutes “wherein the token prediction model comprises a model specific to the first attention head” because the limitation sounds like there are a plurality of prediction models connected to an attention layer. Is the token prediction model an ensemble of a plurality of models that receive each attention from the attention head respectively? Or is it just a single model that receives a plurality of attentions from a plurality of attention heads and generates predictions corresponding to each head?
For purpose of examination, the examiner interprets the claim to mean: The token prediction model is a single model that receives a plurality of attentions from a plurality of attention heads and generates corresponding predictions.
Claim 22 is a method claim which recites the same feature as claim 7 and rejected under the same rationale.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6, 9-10, 16-21 and 24-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: Claim 1 recites A processing system for machine learning, comprising: at least one memory having executable instructions stored thereon; and one or more processors configured to execute the executable instructions. Therefore, it is directed to the statutory category of a machine.
2A Prong 1:
generate,
identify,
generate,
generate an inference based on the second attention map and the second subset of tokens. (mental process of judgment – inferencing, which can be done in the human mind)
2A Prong 2:
A processing system for machine learning, comprising: at least one memory having executable instructions stored thereon; and one or more processors configured to execute the executable instructions in order to cause the processing system to: (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
generate, via a first attention layer of a machine learning model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
identify, using a token prediction model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
generate, via the second attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere generic computer functions that are implemented to perform the disclosed abstract idea above.
2B:
A processing system for machine learning, comprising: at least one memory having executable instructions stored thereon; and one or more processors configured to execute the executable instructions in order to cause the processing system to: (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
generate, via a first attention layer of a machine learning model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
identify, using a token prediction model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
generate, via the second attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are generic computer functions that are implemented to perform the disclosed abstract idea above.
Claim 16 is a method claim which recites the same feature as claim 1 and rejected under the same rationale.
Regarding claim 2,
Step 1: A machine, as above.
2A Prong 1: The processing system of Claim 1,
2A Prong 2: The processing system of Claim 1, wherein the one or more processors are further configured to generate, by a prior attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
2B: The processing system of Claim 1, wherein the one or more processors are further configured to generate, by a prior attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component, See MPEP 2106.05(f))
Claim 17 is a method claim which recites the same feature as claim 2 and rejected under the same rationale.
Regarding claim 3,
Step 1: A machine, as above.
2A Prong 1: The processing system of Claim 1, wherein to generate the inference based on the second attention map and the second subset of tokens,
generate the inference based on the concatenated attention map. (mental process of judgment – inferencing, which can be done in the human mind)
2A Prong 2: the one or more processors are configured to cause the processing system to: (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
2B: the one or more processors are configured to cause the processing system to: (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
Claim 18 is a method claim which recites the same feature as claim 3 and rejected under the same rationale.
Regarding claim 4,
Step 1: A machine, as above.
2A Prong 1: concatenate the second attention map and the second subset of tokens into a concatenated attention map; (mental process of evaluation and judgment – inferencing and combining data, which can be done with the aid of pen and paper)
generate,
identify,
generate,
2A Prong 2: The processing system of Claim 1, wherein the one or more processors are further configured to cause the processing system to: (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
generate, via a third attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
identify, using the token prediction model (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
generate, via the fourth attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
2B: The processing system of Claim 1, wherein the one or more processors are further configured to cause the processing system to: (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
generate, via a third attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
identify, using the token prediction model (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
generate, via the fourth attention layer of the machine learning model (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
Claim 19 is a method claim which recites the same feature as claim 4 and rejected under the same rationale.
Regarding claim 5,
Step 1: A machine, as above.
2A Prong 1: The processing system of Claim 1, wherein to identify the first subset of tokens more relevant to the second attention layer of the machine learning model,
calculate a relevance score of each token based on a prediction of the second attention map generated by the token prediction model; and (a mathematical calculation – See spec [0064])
select k tokens with highest relevance scores for inclusion in the first subset of tokens. (mental process of evaluation – selecting the most relevant data, which can be done in the human mind)
2A Prong 2: the one or more processors are configured to cause the processing system to: (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
2B: the one or more processors are configured to cause the processing system to: (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
Claim 20 is a method claim which recites the same feature as claim 5 and rejected under the same rationale.
Regarding claim 6,
Step 1: A machine, as above.
2A Prong 1: The processing system of Claim 5, wherein k is selected based on an attention ratio defining a percentage of total input tokens to be processed by the second attention layer of the machine learning model. (mental process of judgment – deciding how many tokens to select, which can be done in the human mind)
2A Prong 2: This judicial exception is not integrated into a practical application.
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claim 21 is a method claim which recites the same feature as claim 6 and rejected under the same rationale.
Regarding claim 9,
Step 1: A machine, as above.
2A Prong 1: The processing system of Claim 1,
2A Prong 2: wherein the token prediction model comprises a convolutional model configured to (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
2B: wherein the token prediction model comprises a convolutional model configured to (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
Claim 24 is a method claim which recites the same feature as claim 9 and rejected under the same rationale.
Regarding claim 10,
Step 1: A machine, as above.
2A Prong 1: The processing system of Claim 1,
2A Prong 2: wherein the token prediction model comprises a model configured to (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
2B: wherein the token prediction model comprises a model configured to (mere instructions to apply an exception using a generic computer component MPEP 2106.05(f))
Claim 25 is a method claim which recites the same feature as claim 10 and rejected under the same rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 10, 16, 18 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (“GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation”, 09 April 2024, hereinafter ‘Xu 2024’) in view of Xu et al. (“No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling”, 2023, hereinafter ‘Xu 2023’).
Regarding claim 1, Xu 2024 teaches:
A processing system for machine learning, comprising: at least one memory having executable instructions stored thereon; and one or more processors configured to execute the executable instructions in order to cause the processing system to: ([Xu, 2024, 4. Experiments, lines 1-14] shows that NVIDIA A6000 GPU and PyTorch are used to implement the model, which inherently include memories and processors)
generate, via a first attention layer of a machine learning model, a first attention map based on an input into the machine learning model; ([Xu 2024, page 89, Figure 3] The attention map computed in the Multi-Head Self-Attention layer (MHSA) is used to estimate the importance score for each image token)
identify, using a token prediction model, a first subset of tokens more relevant to a second attention layer of the machine learning model and a second subset of tokens less relevant to the second attention layer of the machine learning model, based on the first attention map; ([Xu 2024, page 89, Figure 3 and left col, lines 1-10] The attention map generated by MHSA layer is input to Token Selection to drop less important tokens. The selected tokens
X
p
are propagated. [page 88, right col, 3.2. Efficient token propagation, lines 1-36] discloses measuring regeneration difficulty score and broadcasting ability score to identify less relevant tokens. [page 90, left col, 3.2.3 Token summarization, line 1 – right col, line 19] Token propagation procedure is implemented immediately after the MHSA module on a layer-by-layer basis. The propagated tokens
X
p
are used to generate summarized tokens
X
s
(first subset of tokens))
generate, via the second attention layer of the machine learning model, a second attention map based on the first subset of tokens; ([Xu 2024, page 89, Figure 3] and [page 90, left col, 3.2.3 Token summarization, line 1 – right col, line 19] Token propagation procedure is implemented immediately after the MHSA module on a layer-by-layer basis (Token propagation-FFN-Classifier). The propagated tokens
X
p
are used to generate summarized tokens
X
s
(first subset of tokens). After reducing the number of tokens, the proportional attention with softmax is introduced to generate another attention map
A
(i.e., second attention map), and in addition to proportional attention, the attention map is refined by filtering out trivial attention values. The filtered attention map is the second attention map)
However, Xu does not specifically disclose:
generate an inference based on the second attention map and the second subset of tokens.
Xu 2023 teaches:
generate an inference based on the second attention map and the second subset of tokens. ([Xu 2023, page 4, Fig. 3] shows put back the idle tokens into the attended tokens to generate output tokens. [page 5, 3.2 Token Selection and Idling, line 1 – page 6, line 17] Image tokens with to -K attention scores towards the [CLS] tokens are selected and named the Selected tokens
X
s
and it involves in the MHSA (attention map generation) calculation of [page 5, lines 1-7]. The unselected image tokens remain the same throughout the layer and are called Idle tokens
X
I
(i.e., second subset of tokens). At the end of the layer, the Idle tokes are concatenated back to the feature map based on the corresponding indices I. [page 4, 3 Methods, lines 1-9], [page 8, Table 1] and [page 14, lines 1-3] show that the image classification task (inference) is performed using the IdleViT)
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to use the method of reusing the unselected tokens to generate final outputs of Xu 2023 to implement the token selection method for efficient inferencing of the present invention. The suggestion and/or motivation for doing so is to improve the accuracy of the Vision Transformer by mitigating the negative impact of improper pruning in the early stages [Xu 2023, Abstract, lines 13-15].
Claim 16 is a method claim which recites the same feature as claim 1 and rejected under the same rationale.
Regarding claim 3, Xu 2024 in view of Xu 2023 teaches:
The processing system of Claim 1, wherein to generate the inference based on the second attention map and the second subset of tokens, the one or more processors are configured to cause the processing system to: ([Xu 2023, page 4, Fig. 3] shows put back the idle tokens into the attended tokens to generate output tokens. [Xu 2023, page 4, 3 Methods, lines 1-9], [page 8, Table 1] and [page 14, lines 1-3] show that the image classification task (inference) is performed using the IdleViT)
concatenate the second attention map and the second subset of tokens into a concatenated attention map; and ([Xu 2023, page 4, Fig. 3] shows put back the idle tokens into the attended tokens to generate output tokens. [page 5, 3.2 Token Selection and Idling, line 1 – page 6, line 17] Image tokens with to -K attention scores towards the [CLS] tokens are selected and named the Selected tokens
X
s
and it involves in the MHSA (attention map generation) calculation of [page 5, lines 1-7]. The unselected image tokens remain the same throughout the layer and are called Idle tokens
X
I
(i.e., second subset of tokens). At the end of the layer, the Idle tokes are concatenated back to the feature map based on the corresponding indices I)
generate the inference based on the concatenated attention map. ([Xu 2023, page 4, 3 Methods, lines 1-9], [page 8, Table 1] and [page 14, lines 1-3] show that the image classification task (inference) is performed using the IdleViT)
Claim 18 is a method claim which recites the same feature as claim 3 and rejected under the same rationale.
Regarding claim 10, Xu 2024 teaches:
The processing system of Claim 1, wherein the token prediction model comprises a model configured to generate normalized attention maps for a plurality of attention heads in the machine learning model. ([Xu, page 88, 3.1. Preliminaries, lines 3-16] In the MHSA layer, a feature map is normalized to generate a layer-normalized feature map, and a softmax activation is performed to generate an attention map using the equation (1). [Xu 2024, page 88, right col, lines 8-33] The equation (3) shows a plurality of Attentions
A
i
,
j
are generated from multiple heads of the multi-head self-attention layer)
Claim 25 is a method claim which recites the same feature as claim 10 and rejected under the same rationale.
Claims 2, 4, 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Xu 2024 in view of Xu 2023 and further in view of Fayyaz et al. (US 20230153379 A1, hereinafter ‘Fayyaz’).
Regarding claim 2, Xu 2024 in view of Xu 2023 teaches:
The processing system of Claim 1.
However, Xu 2024 in view of Xu 2023 do not specifically disclose:
wherein the one or more processors are further configured to generate, by a prior attention layer of the machine learning model, an attention map based on the input into the machine learning model, wherein an input into the first attention layer of the machine learning model comprises the attention map generated by the prior attention layer.
Fayyaz teaches:
wherein the one or more processors are further configured to generate, by a prior attention layer of the machine learning model, an attention map based on the input into the machine learning model, wherein an input into the first attention layer of the machine learning model comprises the attention map generated by the prior attention layer. (According to the instant spec [0024] and a drawing [Fig. 2], the prior attention layer is interpreted as the attention layer of a previous transformer module. [Fayyaz, Fig. 1] and [0030]-[0032] All of the encoder blocks 128, 130 … 132 include respective modified attention components 134 … 138 which includes attention component 140, scoring component 142 ([0033] generates scores assigned to the image tokens 114), and sampling component 144. The second encoder block 130 receives output of the first encoder block 128 to generate second stage hidden state information)
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to use the method of using a plurality of transformers to repeatedly process input data of Fayyaz to implement the token selection method for efficient inferencing of the present invention. The suggestion and/or motivation for doing so is to improve the efficiency of the Vision Transformer based inference system by removing as much non-essential portions of the image data as possible [Fayyaz, 0061].
Claim 17 is a method claim which recites the same feature as claim 2 and rejected under the same rationale.
Regarding claim 4, Xu 2024 in view of Xu 2023 teaches:
The processing system of Claim 1, wherein the one or more processors are further configured to cause the processing system to: ([Xu, 2024, 4. Experiments, lines 1-14] shows that NVIDIA A6000 GPU and PyTorch are used to implement the model)
concatenate the second attention map and the second subset of tokens into a concatenated attention map; ([Xu 2023, page 4, Fig. 3] shows put back the idle tokens into the attended tokens to generate output tokens. [page 5, 3.2 Token Selection and Idling, line 1 – page 6, line 17] Image tokens with to -K attention scores towards the [CLS] tokens are selected and named the Selected tokens
X
s
and it involves in the MHSA (attention map generation) calculation of [page 5, lines 1-7]. The unselected image tokens remain the same throughout the layer and are called Idle tokens
X
I
(i.e., second subset of tokens). At the end of the layer, the Idle tokes are concatenated back to the feature map based on the corresponding indices I)
However, Xu 2024 in view of Xu 2023 do not specifically disclose:
generate, via a third attention layer of the machine learning model, a third attention map based on the concatenated attention map;
identify, using the token prediction model, a third subset of tokens more relevant to a fourth attention layer of the machine learning model and a fourth subset of tokens less relevant to the fourth attention layer of the machine learning model based on the first attention map; and
generate, via the fourth attention layer of the machine learning model, a fourth attention map based on the third subset of tokens, wherein the inference is generated further based on the fourth attention map and the fourth subset of tokens.
Fayyaz teaches:
generate, via a third attention layer of the machine learning model, a third attention map based on the concatenated attention map; ([Fayyaz, Fig. 1] and [0030]-[0032] All of the encoder blocks 128, 130 … 132 include respective modified attention components 134 … 138 which includes attention component 140 (generates the third attention map), scoring component 142 ([0033] generates scores assigned to the image tokens 114), and sampling component 144. The second encoder block 130 receives output of the first encoder block 128 to generate second stage hidden state information)
identify, using the token prediction model, a third subset of tokens more relevant to a fourth attention layer of the machine learning model and a fourth subset of tokens less relevant to the fourth attention layer of the machine learning model based on the first attention map; and ([Fayyaz, Fig. 1] and [0030]-[0032] All of the encoder blocks 128, 130 … 132 include respective modified attention components 134 … 138 which includes attention component 140, scoring component 142 ([0033] generates scores assigned to the image tokens 114), and sampling component 144. The second encoder block 130 receives output of the first encoder block 128 to generate second stage hidden state information. It is obvious for someone who knows the art to combine the method of putting back the idle tokens (fourth subset of tokens) to generate concatenated output (fourth attention) of Xu 2023 and the method of utilizing previous attention transformer output as an input of another transformer of Fayyaz, as Fayyaz discloses connecting a plurality of transformers and Xu 2023 implies that there are another layers following the output tokens [Xu, 2023, page 4, Fig. 3])
generate, via the fourth attention layer of the machine learning model, a fourth attention map based on the third subset of tokens, wherein the inference is generated further based on the fourth attention map and the fourth subset of tokens. ([Fayyaz, Fig. 1] and [0030]-[0032] All of the encoder blocks 128, 130 … 132 include respective modified attention components 134 … 138 which includes attention component 140, scoring component 142 ([0033] generates scores assigned to the image tokens 114), and sampling component 144. The second encoder block 130 receives output of the first encoder block 128 to generate second stage hidden state information. It is obvious for someone who knows the art to combine the method of putting back the idle tokens to generate concatenated output of Xu 2023 and the method of utilizing previous attention transformer output as an input of another transformer of Fayyaz, as Fayyaz discloses connecting a plurality of transformers and Xu 2023 implies that there are another layers following the output tokens [Xu, 2023, page 4, Fig. 3])
Claim 19 is a method claim which recites the same feature as claim 4 and rejected under the same rationale.
Claims 5-9 and 20-24 are rejected under 35 U.S.C. 103 as being unpatentable over Xu 2024 in view of Xu 2023 and further in view of Wang et al. (“PREDICTIVE ATTENTION TRANSFORMER: IMPROVING TRANSFORMER WITH ATTENTION MAP PREDICTION”, 2021, hereinafter ‘Wang’).
Regarding claim 5, Xu 2024 teaches:
The processing system of Claim 1, wherein to identify the first subset of tokens more relevant to the second attention layer of the machine learning model, the one or more processors are configured to cause the processing system to: ([Xu, 2024, 4. Experiments, lines 1-14] shows that NVIDIA A6000 GPU and PyTorch are used to implement the model. [Xu, page 89, left col, lines 1-10] N – P tokens with the largest Γ× Ψ score values are kept)
calculate a relevance score of each token [page 88, right col, 3.2. Efficient token propagation, lines 1-36] discloses measuring regeneration difficulty score and broadcasting ability score to identify less relevant tokens. [Xu, page 89, left col, lines 1-10] N – P tokens with the largest Γ× Ψ score values are kept)
select k tokens with highest relevance scores for inclusion in the first subset of tokens. ([Xu, page 89, left col, lines 1-10] N – P tokens with the largest Γ× Ψ score values are kept)
However, Xu 2024 in view of Xu 2023 do not specifically disclose:
calculate a relevance score based on a prediction of the second attention map generated by the token prediction model;
Wang teaches:
calculate a relevance score based on a prediction of the second attention map generated by the token prediction model; ([Wang, page 5, lines 1 – 19] An attention map is predicted using the convolution-based attention prediction model. The predicted attention map
A
is used by the value projection layer to generate value representation
H
i
which is interpreted as the score)
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to use the method of using a convolutional attention map prediction model of Wang to implement the token selection method for efficient inferencing of the present invention. The suggestion and/or motivation for doing so is to improve the inference performances of the Transformer based inference system [Wang, page 6, Results].
Claim 20 is a method claim which recites the same feature as claim 5 and rejected under the same rationale.
Regarding claim 6, Xu 2024 in view of Xu 2023 teaches:
The processing system of Claim 5, wherein k is selected based on an attention ratio defining a percentage of total input tokens to be processed by the second attention layer of the machine learning model. ([Xu 2023, page 4, Table 4 and paragraphs above and below the Table] The base keep ratio is set to be 0.9, 0.8 … 0.5 when testing the token selection method. This indicates that the number of surviving tokens is pre-set based on the base keep ratio)
Claim 21 is a method claim which recites the same feature as claim 6 and rejected under the same rationale.
Regarding claim 7, Xu 2024 teaches:
The processing system of Claim 1, wherein the first attention map and the second attention map comprise attention maps for a first attention head in the machine learning model ([Xu 2024, page 88, right col, lines 8-33] The equation (3) shows aggregating a plurality of Attentions
A
i
,
j
from multiple heads of the multi-head self-attention layer including the first attention head)
Xu 2024 in view of Xu 2023 do not specifically disclose:
wherein the token prediction model comprises a model specific to the first attention head.
Wang teaches:
wherein the token prediction model comprises a model specific to the first attention head. ([Wang, page 4, the last paragraph] The output channel is also set to be K, so the attention maps of all heads can be generated jointly. This indicates that predictions are generated for each attention head respectively)
Claim 22 is a method claim which recites the same feature as claim 7 and rejected under the same rationale.
Regarding claim 8, Xu 2024 teaches:
The processing system of Claim 7, wherein to identify the first subset of tokens and the second subset of tokens, the one or more processors are configured to cause the processing system to predict relevant tokens based on a concatenation of the attention maps for the first attention head and attention maps for one or more additional attention heads in the machine learning model. ([Xu, 2024, 4. Experiments, lines 1-14] shows that NVIDIA A6000 GPU and PyTorch are used to implement the model. [Xu 2024, page 88, right col, lines 8-33] The equation (3) shows aggregating a plurality of Attentions
A
i
,
j
from multiple heads of the multi-head self-attention layer.
γ
i
indicates a more important token
x
i
)
Claim 23 is a method claim which recites the same feature as claim 8 and rejected under the same rationale.
Regarding claim 9,
The processing system of Claim 1.
However, Xu 2024 in view of Xu 2023 do not specifically disclose:
wherein the token prediction model comprises a convolutional model configured to generate a predicted attention map for the second attention layer of the machine learning model based on an input of a first attention map generated by the first attention layer of the machine learning model.
Wang teaches:
wherein the token prediction model comprises a convolutional model configured to generate a predicted attention map for the second attention layer of the machine learning model based on an input of a first attention map generated by the first attention layer of the machine learning model. ([Wang. Figure 1], [page 4, lines 1-11 and 3.3 Convolution-Based Attention Prediction, lines 1-12] The Convolution-based Attention Map Prediction receives an Attention map generated by the Attention Map Generator, predicts attentions, and provides attentions to Block 2)
Claim 24 is a method claim which recites the same feature as claim 9 and rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Rao et al., “DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification” (This prior art is pertinent because it discloses attention layer – prediction – attention layer structure in Fig. 2)
Rao et al., “Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks” (This prior art is pertinent because it discloses attention layer – prediction – attention layer structure in Fig. 2)
Bolya et al., “TOKEN MERGING: YOUR VIT BUT FASTER” (This prior art is pertinent because it discloses classifying input tokens into two groups)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUN KWON whose telephone number is (571)272-2072. The examiner can normally be reached Monday – Friday 8:00AM – 5:00PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUN KWON/Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127
