DETAILED ACTION
This action is written in response to the remarks and amendments dated 3/27/26. This action is made final. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The Applicants argue that the previous art of record does not anticipate or render obvious the claims as currently amended. The Examiner provides updated prior art rejections below necessitated by the current amendments. Additional arguments are also addressed below.
Subject Matter Eligibility
In determining whether the claims are subject matter eligible, the examiner has considered and applied the 2019 USPTO Patent Eligibility Guidelines, as well as guidance in the MPEP chapter 2106. The examiner finds that the combination of steps recited in the independent claims cannot be practically performed as a mental process.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
Jiang (US 2024/0428576 A1)
Lee (US 2023/0376851 A1)
Luo (Luo, Ziyang, et al. "Analyzing the Implicit Position Encoding Ability of Transformer Decoder." Published at OpenReview.net 28 Sept 2021. 15 pages.)
Lin (Lin et al., “A survey of transformers”, CoRR, submitted June 8, 2021, arXiv:2106.04554v1, 40 pages. Cited by Applicant in IDS dated 10/22/25.)
Shazeer (US 2020/0342316 A1)
Siohan (US 2023/0186198 A1)
Claims 1-2, 5, 7-9, 11-13, 16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lee and Luo.
Regarding claims 1 and 12 and 20, Lee discloses a method (and a related system and non-transitory computer-readable media) performed by one or more computers, the method comprising:
receiving an input sequence comprising a respective input token at each of a plurality of input positions; and
Fig. 3, (reproduced below), “input 305”.
PNG
media_image1.png
888
556
media_image1.png
Greyscale
processing the input sequence using a neural network to generate a network output for a machine learning task, wherein:
Id. “output 380”.
the neural network comprises a plurality of layers that comprise one or more global attention layers and a plurality of local attention layers,
Id. “local attention” and “global attention”.
the plurality of layers are arranged in a sequence and each of the one or more global attention layers is preceded by a respective subset of the plurality of local attention layers in the sequence of layers,
Id. “local attention 330” and “global attention 360”.
each global attention layer receives a respective hidden state for each of the plurality of input positions and updates a respective hidden state for each of at least one of the plurality of input positions based on applying a global attention mechanism that, for each of at least one of the plurality of input positions, attends over all of the plurality of input positions preceding or equal to the input position in the input sequence,
Id.
‘hidden state’ :: learned weight values within each neuron of each layer of the depicted network. See eg [0026] and [0030] for additional description of transformer layers within a neural network model.
each local attention layer receives a respective hidden state for each of the plurality of input positions and updates a respective hidden state for each of at least one of the plurality of input positions based on applying a local attention mechanism that, for each of at least one of the plurality of input positions, attends only over a set of local input positions that are within a local window of the input position in the input sequence,
[0037] “In some aspects, local attention (also referred to as sliding window attention) limits the attention range to the vicinity of query locations. That is, key abstraction may be performed with the whole abstraction range, and the query abstraction may be performed using a location-dependent abstraction function…”
the local attention mechanism applied by each local attention layer uses position encoding, and
[0048] “Because transformer-based models generally contain no recurrence and no convolution, in some aspects, some information about the relative or absolute position of the tokens in the sequence is injected in order for the model to make use of the order of the sequence. This may be referred to in some aspects as positional embedding (e.g., referred to in some aspects as Pl for local positional embeddings and Pg for global positional embeddings, and indicated by embedding functions 207 and 209, respectively, in FIG. 2 and embedding functions 314 and 344, respectively, in FIG. 3 ).” (Emphasis added.)
the global attention mechanism applied by each global attention layer does not use … position encoding.
As described at [0048] (reproduced above) and [0067], (reproduced below) global attention layers do not use the same position encodings as local attention layers, although they (optionally) may use a distinct position encoding pg.
[0067] “Note that unlike the local positional embeddings, Pl, the global positional embeddings Pg are sized R N/L×1×D consistent with the size of the slice embeddings.”
Luo discloses the following further limitation which Lee does not disclose wherein:
the global attention mechanism applied by each global attention layer does not use any position encoding.
P. 5, “Basic Model. Our basic model is an 8-layer Transformer Decoder with 768 embedding size, 3072 feedforward layer hidden size, 12 attention heads and GELU activation function (Hendrycks & Gimpel, 2020), which is a smaller version of GPT and has 95M trainable parameters for English model and 77.5M for Chinese model.5 We find that if we use a standard 12-layer GPT, the number of trainable parameters will be higher than the number of tokens in the WikiText-103 dataset. This has a risk to cause over-fitting, so we choose to use an 8-layer model. We do not use any position encodings in this model and denote it as No-PE. All variants in our experiments are based on it.” (Emphasis added.)
At the time of filing, it would have been obvious to a person of ordinary skill to combine attention mechanisms without position encoding (as taught by Luo) with the Lee system because—as demonstrated by Luo—this approach can yield superior performance. (See pp. 6-7, sec. 5.3, “Results and analysis”.) This technique is applicable to both global or local attention layers, and would yield predictable results in either case.
Regarding independent claims 12 and 20, Lee also discloses the recited computer components and “one or more storage devices” / “non-transitory computer storage media”, see eg [0006].
Regarding claims 2 and 13, Lee discloses the further limitation wherein each of the one or more global attention layers is preceded by a fixed number of respective local attention layers in the sequence of layers.
Fig. 3 (reproduced supra): ‘global attention 360’ is preceded by one ‘local attention 330’ layer.
Regarding claims 5 and 16, Lee discloses the further limitation wherein the plurality of layers further comprise one or more dense feedforward layers.
[0030] “In some aspects, transformer layers in a neural network model cam include a multi-head self-attention sublayer followed by a feed-forward network”. The Examiner interpret ‘dense’ as encompassing any layer that is not pruned; Lee makes no mention of pruning.
See also [0058] “fully-connected feed-forward network”.
Regarding claims 7-8, its further limitation is an obvious extension of the Lee disclosure. Their further limitation recites wherein a number of input positions in the local window is less than or equal to 1.0% of a number of input positions in the input sequence. The Examiner notes that there are only two possibilities for local window size: ≤ 0 or > 0. This is a tunable parameter that can trivially be set, eg by trial-and-error, depending on the input size and the relative size of the patterns which the system engineers are trying to identify within the input data. At the time of filing, it would have been obvious to a person of ordinary skill to choose from among the two possibilities identified when tuning this parameter.
Regarding claims 9 and 18, Lee discloses the further limitation wherein the respective input tokens at the plurality of input positions comprise tokens representing one or more of
[0043] “Therefore, aspects of the present disclosure may be more suitable for modeling highly non-stationary data, such as natural language text data for which a locality assumption does not hold.”
Regarding claim 11, its further limitation is an obvious extension of the Lee system. Its further limitation recites wherein the machine learning task comprises a long context task that requires processing a long input sequence comprising at least one million input tokens.
Although this limitation does not seem to be explicitly disclosed in Lee, the reference does consider scalability extensively, as illustrated in the passage below.
[0036] “Multi-Scale Multi-Range Attention Although some previous abstractive attention and non-attention approaches have achieved sub-quadratic complexity (and even linear complexities for some methods), these prior approaches generally come at the cost of degraded performance (e.g., reduced accuracy) on benchmarks. However, the efficient transformer-based model architectures described herein leverage multi-scale attention by combining local attention and global attention and provide significant accuracy improvements (often outperforming conventional architectures) while still maintaining the efficiency benefits.”
Scalability becomes a concern when training the network on a large corpus of text documents (eg a large collections of websites or books) or even fairly small corpora of audio, video or other multimedia data. One million tokens wasn’t especially large, even at the time of filing. At the time of filing, it would have been obvious to a person of ordinary skill to apply the Lee system to large corpora of data so that insight could be gained therefrom.
Claims 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, Luo and Jiang.
Regarding claims 3 and 14, Jiang discloses the further limitation which Lee/Luo do not disclose wherein the fixed number is three.
[0115] “The switch component 530 may then select the attention scheme (e.g., the number of local attentions, such as ranging from zero to three) to apply based on these comparisons. Although the illustrated example depicts three local attention blocks 710, in some aspects, the architecture 700 may include any number of (optional) local attention blocks.”
At the time of filing, it would have been obvious to a person of ordinary skill to apply three attention layers (as taught by Jiang) in combination with the Lee/Luo system because this architecture may improve the model’s ability to identify patterns at differing scales.
Claims 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, Luo and Shazeer.
Regarding claims 4 and 15, Shazeer discloses the following further limitation which Lee/Luo do not disclose wherein the plurality of layers further comprise one or more Mixture of Experts (MoE) layers.
[0058] “While not shown in FIG. 1, in some cases, to increase the computational capacity of the decoder neural network 150 without excessive increases in processing time or computational cost, the decoder neural network 150 can include one or more mixture of experts layers.”
At the time of filing, it would have been obvious to a person of ordinary skill to include a MoE layer in the neural network architecture of Lee/Luo because—as noted in the passage above—this can increase the computational capacity of the network.
Claims 6 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, Luo and Lin.
Regarding claims 6 and 17, Lin discloses the following further limitation which Lee/Luo do not disclose wherein the position encoding comprises a Rotary Position Embedding (RoPE) position encoding.
P. 22, “Roformer [123] uses Rotary Position Embedding (RoPE) to represent the position of a token by multiplying the affine-transformed embedding of the 𝑡-th input 𝑥𝑡 by a rotatory matrix”.
At the time of filing, it would have been obvious to a person of ordinary skill to apply RoPE to the Lee/Luo system for the particular advantage noted by Lin: “The key advantage of this formulation is that the induced representation is translation invariant” (p. 22).
Claims 10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, Luo and Siohan.
Regarding claims 10 and 19, Siohan discloses the following further limitation which Lee/Luo do not disclose wherein the machine learning task comprises a multi-modal task that requires processing two or more of: audio data, image data, or text data.
[0027] “The content feed 112 may be an audio feed 218 (i.e., audio data 218 such as audio content, an audio signal, or audio stream), a visual feed 217 (i.e., image data 217 such as video content, a video signal, or video stream), or some combination of both (e.g., also referred to as an audio-visual feed, an audio-visual signal, or an audio-visual stream).”
At the time of filing, it would have been obvious to a person of ordinary skill to apply the system of Lee to audio-visual data (as taught by Siohan) because this would enable the model to identify useful insight from this kind of data. Both Lee and Siohan pertain to attention neural networks. (See eg Siohan [0005] “Here, the action item classification model may include a pre-trained Bidirectional Encoder representations from Transformers (BERT) model or an extended transformer construction (ETC) model having a global-local attention mechanism.”)
Additional Relevant Prior Art
The following references were identified by the Examiner as being relevant to the disclosed invention, but are not relied upon in any particular prior art rejection:
Parisotto discloses a gated attention neural network system for multi-modal streaming data (see eg [0016] and [0029]). (US 2022/0366218 A1)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Vincent Gonzales whose telephone number is (571) 270-3837. The examiner can normally be reached on Monday-Friday 7 a.m. to 4 p.m. MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
Information regarding the status of an application may be obtained from the USPTO Patent Center.
/Vincent Gonzales/Primary Examiner, Art Unit 2124
Prosecution Insights