Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant argues,
With respect to patent eligible subject matter, Applicant's arguments filed 4/16/2026 have been fully considered but they are not persuasive.
Using a suitable distribution of weights over the various dimensions when combining input tokens can allow for more dynamic and complex combinations of meaning, leading to the creation of machine learning models that are more effective in terms of identifying user intents or performing other functions and may be used to create modified attention mechanisms to be used by machine learning models. Applicant's Specification, par. [0034]. Thus, as in Enfish, the claimed features are directed to a specific improvements to computer functionality, namely providing modified attention mechanisms used by machine learning models, thereby improving the functioning and output of the machine learning models.
MPEP § 2106.04(a)(2)(III)(A) states:
Claims do not recite a mental process when they do not contain limitations that can practically be performed in the human mind, for instance when the human mind is not equipped to perform the claim limitations.
Remarks 13.
Examiner respectfully disagrees because of the breadth of the claim. The way the claims are currently amended this weight vector could be as small as two elements and it could be used to do multiplication on a 2x2 matrix of input tokens. That is something a person can do in their head. Therefore, the claimed algorithm is a mental concept.
Applicant argues,
The Office fails to perform an analysis of Claim 1 as a whole and merely asserts that certain additional elements are generic computer parts and that one operation represents insignificant extra-solution activity. Office Action, page 2.
Remarks 14-15.
Examiner respectfully disagrees. The legal question is whether “whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer.” MPEP 2106.05(f). Here the processor and memory are additional elements “that amount to nothing more than an instruction to apply the abstract idea using a generic computer…” Id.
Applicant argues,
In reality, Claim 1 includes obtaining a specific input (multiple tokens), processing the tokens using a machine learning model in a specific manner using specific operations, and generating a specific output, which improves the functioning of a computer. The Applicant's claims are directed to solving the above-noted specific technical problem.
Remarks 15.
This amendment is using a computer to execute an algorithm. The algorithm doesn’t change the functioning of the computer and there isn’t a mention of a technical field in the claims.
With respect to the prior art rejections, Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea of a mental concept or mathematical relationship without significantly more. The claims recite the abstract idea of processing input using attention, generating an output embedding vectors, generating a query representation, generating vectors, applying a softmax function, determining a distance between a query and a representations of a token, performing dimensional-level mixing, performing token-level mixing, performing first and second attention and using attention heads to output vectors for each token. This judicial exception is not integrated into a practical application because the additional elements of non-transitory machine readable media and processors are generic computer parts, and obtaining input data is insignificant extra-solution activity. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements of non-transitory machine readable media and processors are generic computer parts, and obtaining input data is insignificant extra-solution activity.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-11 and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over DaViT: Dual Attention Vision Transformers by Ding et al and Exploring Self-attention for Image Recognition by Zhao et al.
Claims 4 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over DaViT: Dual Attention Vision Transformers by Ding et al, Exploring Self-attention for Image Recognition by Zhao et al and Attention Is All You Need by Vaswani et al.
Ding teaches claims 1, 9 and 17. A method comprising:
obtaining an input containing multiple tokens; (Ding abs “We propose approaching the problem from an orthogonal angle: exploiting self-attention mechanisms with both “spatial tokens” and “channel tokens”.” Each token gets parsed into its Q, K, V based on channel and space.)
processing the input using a machine learning model, wherein processing the input comprises performing attention over both (i) multiple dimensions of the tokens contained in the input and (ii) (Ding p. 2 last paragraph “we introduce ‘channel tokens’ by applying self-attention to the transpose of the token matrix…” a Channel token is the result of processing the token matrix by applying self attention. The transformer in Ding is a machine learning model. This is shown as spatial window attention in Fig. 3b.) multiple dimensions of embedding vectors used to represent the tokens contained in the input so that different dimensions of each of at least some of the tokens are weighted differently; and (Ding fig. 3c channel group attention in 3c. The different weights are taught as “attention scores between channels”.)
PNG
media_image1.png
508
940
media_image1.png
Greyscale
generating an output embedding vector for a query token of the multiple tokens based on the attention. (Ding p. 6 eq. 1 attention head i of the input feature generates an output embedding A(Q,K,V) for each input token based on attention. Qi is the query token. Also the projection layers and output of the dual attention block in Ding fig. 3 shows an output embedding vector for a query token based on the attention modules.)
Ding doesn’t teach a weight vector of token pairs.
However, Zhao teaches generating, for each pair of tokens, a weight vector that includes a separate weight for each of the multiple dimensions of the embedding vectors. (Zhao sec. 2 “our primary self attention mechanisms compute a vector attention that adapts to different channels, rather than a shared scalar weight.” The equation below shows the pair of tokens xi, xj and the weight vector α.
PNG
media_image2.png
80
434
media_image2.png
Greyscale
Ding, Zhao and the claims are all attention mechanisms. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use a weight vector instead of a scalar because “[c]ontrolled experiments also indicate that our vectorial operators outperform standard scalar attention.” Zhao sec. 1.
Ding teaches claims 2, 10 and 18. The method of claim 1, wherein performing the attention comprises:
for each of the tokens contained in the input:
generating one of the embedding vectors that represents the token; and (Ding fig. 3 projection output before Q K V in Ding fig. 3.)
generating a query representation by multiplying the embedding vector with a query matrix, a key representation by multiplying the embedding vector with a key matrix, and a value representation by multiplying the embedding vector with a value matrix; and (Ding fig. 3 Q K and V, respectively)
for the query token:
generating attention weight vectors based on the query representation associated with the query token and the key representations associated with the tokens; (Ding. Fig. 3 below, the input PW x P-W)
PNG
media_image3.png
256
338
media_image3.png
Greyscale
applying a softmax function to the attention weight vectors in order to generate modified attention weight vectors, and (Ding fig. 3 softmax of the input Pw x Pw.)
generating weighted value vectors by multiplying the value representations and the modified attention weight vectors, (Ding fig. 3 Pw x Ch from value representations V, below.) the output embedding vector for the query token based on a sum of the weighted value vectors. (Ding fig. 3 below
PNG
media_image4.png
256
338
media_image4.png
Greyscale
PNG
media_image5.png
454
936
media_image5.png
Greyscale
Ding teaches claims 3 and 11. The method of claim 2, wherein at least one of:
each query representation includes a different query vector for each of multiple dimensions; and
each key representation includes a different key vector for each of multiple dimensions. (Ding fig. 3 Q and K for the channel group attention module, the channels including multiple dimensions.)
Ding teaches claims 4 and 12. The method of claim 2, wherein generating the attention weight vectors comprises determining a distance or similarity between (Ding p. 16 “Considering larger
image resolutions and model sizes used, we suspect that the dot products of
self-attention grow large in magnitude in this case, as in [59]. To counteract
this effect, we scale the dot products in our channel attention by 1/√P.”)
Ding doesn’t teach what the dot product is.
However, Vaswani teaches similarity between (i) the query representation associated with the query token and (ii) each of the key representations associated with the tokens. (Vaswani p. 3 “the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.” Vaswani sec. 3.2.1 p. 4 “We compute the dot products of the query with all keys…” Dot product is a similarity and distance between two vectors.)
Ding, Vaswani and the claims all use a soft max to determine attention weights based in Q and K. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to Ding specifically references Vaswani as “[59]” and Vaswani shows that the query and the key are inputs to the dot product taught by Ding.
Ding teaches claims 5 and 13. The method of claim 2, wherein each of the query representations, key representations, and value representations includes or is associated with a dimensional embedding, the dimensional embedding identifying one of the dimensions associated with one of the tokens. (Ding fig. 1 teaches this below. The query and key representations are taught as Q and K in fig. 3 of Ding.)
PNG
media_image6.png
226
952
media_image6.png
Greyscale
Ding teaches claims 6, 14 and 19. The method of claim 2, wherein performing the attention comprises:
performing dimensional-level mixing of features based on a transpose of a token embedding matrix using a first self-attention operation, the token embedding matrix containing the embedding vectors of the tokens; and (Ding p. 2 ”representing the feature of an image patch, we introduce “channel tokens” by applying self-attention to the transpose of the token matrix…”)
performing token-level mixing of context based on a transpose of results of the first self-attention operation using a second self-attention operation. (Ding sec. 3.1 “we alternatively arrange spatial window attention and channel group attention to obtain both local and global features.” This shows that the spatial attention module can take the results from the channel attention module. And both module “apply attention mechanisms on the transpose of patch-level tokens.” Ding sec. 3.3.)
Ding teaches claims 7 and 15. The method of claim 2, wherein:
performing the attention comprises performing dimensional-level mixing of features and token-level mixing of context based on a token embedding matrix using first and second self-attention operations, the token embedding matrix containing the embedding vectors of the tokens; (Ding sec. 3.1 “we alternatively arrange spatial window attention and channel group attention to obtain both local and global features.” This shows that the spatial attention module can take the results from the channel attention module. And both module “apply attention mechanisms on the transpose of patch-level tokens.” Ding sec. 3.3.)
the first and second self-attention operations are performed in parallel; and (Ding sec. 5.4 and table 6, “There are three options with similar computations: (i) window attention first; (ii) channel attention first; and (iii) two types of attention are paralleled [sic] arranged.”)
PNG
media_image7.png
172
984
media_image7.png
Greyscale
a transpose of results of one of the first and second self-attention operations is multiplied by results of another of the first and second self-attention operations to generate final attention weights. (Any time Ding softmaxes output vectors, Ding transposes the K, see eqn. 1 below. So the results will have at least one transpose of results.)
PNG
media_image8.png
172
728
media_image8.png
Greyscale
Ding teaches claims 8, 16 and 20. The method of claim 2, wherein:
the machine learning model comprises multiple attention heads;
each attention head is configured to generate an output embedding vector for each token; and
the method further comprises using the output embedding vectors associated with each token to generate a single output embedding for the token. (Ding eqn. 1 below. “i” is the token. Softmax is the output embedding vector associated with each token.)
PNG
media_image9.png
168
732
media_image9.png
Greyscale
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
Prosecution Insights