Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea of a mental concept or mathematical relationship without significantly more. The claims recite the abstract idea of processing input using attention, generating an output embedding vectors, generating a query representation, generating vectors, applying a softmax function, determining a distance between a query and a representations of a token, performing dimensional-level mixing, performing token-level mixing, performing first and second attention and using attention heads to output vectors for each token. This judicial exception is not integrated into a practical application because the additional elements of non-transitory machine readable media and processors are generic computer parts, and obtaining input data is insignificant extra-solution activity. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements of non-transitory machine readable media and processors are generic computer parts, and obtaining input data is insignificant extra-solution activity.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 5-11 and 13-20 are rejected under 35 U.S.C. 102(a)(1) as being described by DaViT: Dual Attention Vision Transformers by Ding et al.
Ding teaches claims 1, 9 and 17. A method comprising:
obtaining an input containing multiple tokens; (Ding abs “We propose approaching the problem from an orthogonal angle: exploiting self-attention mechanisms with both “spatial tokens” and “channel tokens”.” Each token gets parsed into its Q, K, V based on channel and space.)
processing the input using a machine learning model, wherein processing the input comprises performing attention over both (i) multiple dimensions of the tokens contained in the input and (ii) (Ding p. 2 last paragraph “we introduce ‘channel tokens’ by applying self-attention to the transpose of the token matrix…” a Channel token is the result of processing the token matrix by applying self attention. The transformer in Ding is a machine learning model. This is shown as spatial window attention in Fig. 3b.) multiple dimensions of embedding vectors used to represent the tokens contained in the input so that different dimensions of each of at least some of the tokens are weighted differently; and (Ding fig. 3c channel group attention in 3c. The different weights are taught as “attention scores between channels”.)
PNG
media_image1.png
508
940
media_image1.png
Greyscale
generating an output embedding vector for a query token of the multiple tokens based on the attention. (Ding p. 6 eq. 1 attention head i of the input feature generates an output embedding A(Q,K,V) for each input token based on attention. Qi is the query token. Also the projection layers and output of the dual attention block in Ding fig. 3 shows an output embedding vector for a query token based on the attention modules.)
Ding teaches claims 2, 10 and 18. The method of claim 1, wherein performing the attention comprises:
for each of the tokens contained in the input:
generating one of the embedding vectors that represents the token; and (Ding fig. 3 projection output before Q K V in Ding fig. 3.)
generating a query representation by multiplying the embedding vector with a query matrix, a key representation by multiplying the embedding vector with a key matrix, and a value representation by multiplying the embedding vector with a value matrix; and (Ding fig. 3 Q K and V, respectively)
for the query token:
generating attention weight vectors based on the query representation associated with the query token and the key representations associated with the tokens; (Ding. Fig. 3 below, the input PW x P-W)
PNG
media_image2.png
256
338
media_image2.png
Greyscale
applying a softmax function to the attention weight vectors in order to generate modified attention weight vectors, and (Ding fig. 3 softmax of the input Pw x Pw.)
generating weighted value vectors by multiplying the value representations and the modified attention weight vectors, (Ding fig. 3 Pw x Ch from value representations V, below.) the output embedding vector for the query token based on a sum of the weighted value vectors. (Ding fig. 3 below
PNG
media_image3.png
256
338
media_image3.png
Greyscale
PNG
media_image4.png
454
936
media_image4.png
Greyscale
Ding teaches claims 3 and 11. The method of claim 2, wherein at least one of:
each query representation includes a different query vector for each of multiple dimensions; and
each key representation includes a different key vector for each of multiple dimensions. (Ding fig. 3 Q and K for the channel group attention module, the channels including multiple dimensions.)
Ding teaches claims 5 and 13. The method of claim 2, wherein each of the query representations, key representations, and value representations includes or is associated with a dimensional embedding, the dimensional embedding identifying one of the dimensions associated with one of the tokens. (Ding fig. 1 teaches this below. The query and key representations are taught as Q and K in fig. 3 of Ding.)
PNG
media_image5.png
226
952
media_image5.png
Greyscale
Ding teaches claims 6, 14 and 19. The method of claim 2, wherein performing the attention comprises:
performing dimensional-level mixing of features based on a transpose of a token embedding matrix using a first self-attention operation, the token embedding matrix containing the embedding vectors of the tokens; and (Ding p. 2 ”representing the feature of an image patch, we introduce “channel tokens” by applying self-attention to the transpose of the token matrix…”)
performing token-level mixing of context based on a transpose of results of the first self-attention operation using a second self-attention operation. (Ding sec. 3.1 “we alternatively arrange spatial window attention and channel group attention to obtain both local and global features.” This shows that the spatial attention module can take the results from the channel attention module. And both module “apply attention mechanisms on the transpose of patch-level tokens.” Ding sec. 3.3.)
Ding teaches claims 7 and 15. The method of claim 2, wherein:
performing the attention comprises performing dimensional-level mixing of features and token-level mixing of context based on a token embedding matrix using first and second self-attention operations, the token embedding matrix containing the embedding vectors of the tokens; (Ding sec. 3.1 “we alternatively arrange spatial window attention and channel group attention to obtain both local and global features.” This shows that the spatial attention module can take the results from the channel attention module. And both module “apply attention mechanisms on the transpose of patch-level tokens.” Ding sec. 3.3.)
the first and second self-attention operations are performed in parallel; and (Ding sec. 5.4 and table 6, “There are three options with similar computations: (i) window attention first; (ii) channel attention first; and (iii) two types of attention are paralleled [sic] arranged.”)
PNG
media_image6.png
172
984
media_image6.png
Greyscale
a transpose of results of one of the first and second self-attention operations is multiplied by results of another of the first and second self-attention operations to generate final attention weights. (Any time Ding softmaxes output vectors, Ding transposes the K, see eqn. 1 below. So the results will have at least one transpose of results.)
PNG
media_image7.png
172
728
media_image7.png
Greyscale
Ding teaches claims 8, 16 and 20. The method of claim 2, wherein:
the machine learning model comprises multiple attention heads;
each attention head is configured to generate an output embedding vector for each token; and
the method further comprises using the output embedding vectors associated with each token to generate a single output embedding for the token. (Ding eqn. 1 below. “i” is the token. Softmax is the output embedding vector associated with each token.)
PNG
media_image8.png
168
732
media_image8.png
Greyscale
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over DaViT: Dual Attention Vision Transformers by Ding et al and Attention Is All You Need by Vaswani et al.
Ding teaches claims 4 and 12. The method of claim 2, wherein generating the attention weight vectors comprises determining a distance or similarity between (Ding p. 16 “Considering larger
image resolutions and model sizes used, we suspect that the dot products of
self-attention grow large in magnitude in this case, as in [59]. To counteract
this effect, we scale the dot products in our channel attention by 1/√P.”)
Ding doesn’t teach what the dot product is.
However, Vaswani teaches similarity between (i) the query representation associated with the query token and (ii) each of the key representations associated with the tokens. (Vaswani p. 3 “the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.” Vaswani sec. 3.2.1 p. 4 “We compute the dot products of the query with all keys…” Dot product is a similarity and distance between two vectors.)
Ding, Vaswani and the claims all use a soft max to determine attention weights based in Q and K. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to Ding specifically references Vaswani as “[59]” and Vaswani shows that the query and the key are inputs to the dot product taught by Ding.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
Prosecution Insights