DETAILED ACTION
1. This office action is in response to Application No. 18044842 filed on 06/18/2026. Claims 1-19 are presented for examination and are currently pending. Applicant’s arguments have been carefully and respectfully considered.
Response to Arguments
2. Applicant’s arguments regarding the prior art rejection are moot in view of the
new grounds of rejection. The Examiner is withdrawing the rejections in the previous
Office Action because the Applicant’s amendments necessitated the new grounds of
rejection presented in this Office Action.
However, Huang, Liu, Li and Luo which was used previously in the art rejection is still relevant and applied.
The dependent claims 2-17, which depend directly or indirectly from independent claims 1, 18 and 19 are not patentable because the instant claims are still obvious over the prior art of record.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1, 4, 6, 8-16, 18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. ("Global-local attention network for aerial scene classification." IEEE Access 7 (2019): 67200-67212). in view of Liu et al. ("PiCANet: Pixel-wise contextual attention learning for accurate saliency detection." IEEE Transactions on Image Processing 29 (2020): 6438-6451, Date of Publication: 23 April 2020).
Regarding claim 1, Guo teaches a computing system for performing modeling of dependencies using global self-attention (a novel end-to-end global-local attention network (GLANet) is proposed to capture both global and local information for aerial scene classification, abstract, Fig. 2, pg. 67202),
comprising: one or more; and one or more non-transitory computer-readable media that collectively store processors (an NVIDIA GTX TITAN GPU, pg. 67205, left col., first para.):
a machine-learned model (the proposed global-local attention network can be trained with small training samples, pg. 67201, right col., second para.) configured to receive a model input and process the model input (Given an input image, we can extract the feature map X through the backbone network, Fig. 2, pg. 67202, left col., second para.) to generate a model output (The three predicted labels … of this network, Fig. 2, pg. 67204, left col., last para.),
wherein the machine-learned model (a novel end-to-end global-local attention network (GLANet) is proposed to capture both global and local information for aerial scene classification, abstract, Fig. 2, pg. 67202) comprises:
a content attention layer (Global Attention Branch, Fig. 2) configured to globally attend to content from the model input without taking spatial positions of the content into account (The top part in Fig.2 is a squeeze-excitation module …
that performs global information abstraction (pg. 67202, left col., last para.); The
global branch is introduced to learn the channel-wise weights for feature maps, pg. 67202, right col., last para.); and
a positional attention layer (Local Attention Branch, Fig. 2) configured to, for each respective spatial position in the model input, attend to a neighborhood of spatial positions relative to the respective spatial position (As shown in the bottom part of Fig. 2, the LA branch utilizes two successive residual attention modules, one of which consists of a spatial attention layer and a residual connection. The spatial attention layer learns to apply weighted mask to the input feature maps (pg. 67203, left col., last para.); The two connected residual modules output the
attention maps, which contains spatial information, pg. 67203, right col., last para.);
wherein the content attention layer (Global Attention Branch, Fig. 2) and the position attention layer (Local Attention Branch, Fig. 2) configured to operate in parallel with each other (see Fig. 2), and
wherein the machine-learned model is configured to perform operations comprising: receiving a layer input (pre-trained fully convolutional neural network on ImageNet …, which con-sists of the first five convolution blocks (i.e., Conv1-Conv5 in Fig.2), pg. 67203, left col., second para.) comprising input data that comprises a plurality of content values each associated with one or more context positions (600 x 600 pixels, pg. 67205, left col., last para. The Examiner notes context positions of content are pixels of an image, and each pixel has an associated content value);
generating, by the content attention layer (Global Attention Branch, Fig. 2), one or more output features (After the weight factor acts on the feature map, then a
global attention feature … for the cth channel is obtained via the second GAP layer, pg. 67203, left col., first para.) for each context position based on a global attention operation applied to the content values-independent of the context positions (In the first GAP layer, the values of different channels can be regarded as the
global context information of the image via global average pooling … pg. 67202, last para. Left col., to right col., first para. The Examiner notes the output features for each context position generated based on the global attention is independent of the positions);
and determining a layer-output (softmax layer, Fig. 2), based at least in part on the one or more output features for each context position generated by the content attention layer and the attention map generated for each context position by the positional attention layer (global, local feature vectors and their concatenation in the softmax layer of this network (pg. 67204, left col., last para., Fig. 2); During each training… local, global, and their concatenated features using softmax, abstract).
Guo does not explicitly teach generating, by the positional attention layer, an attention map for each of the context positions based on one or more of the content values associated with the respective context position and a neighborhood of context positions relative to the respective context position, the positional attention layer comprising at least a column-focused attention sublayer that attends to context positions along a column of each respective context position and a row-focused attention sublayer that attends to context positions along a row of each respective context position;
Liu teaches one or more; and one or more non-transitory computer-readable media that collectively store processors (A GTX 1080Ti GPUis used for acceleration, pg. 6444, right col., second para.; … only use the first four decoding modules to save GPU memory, pg. 6449, left col., first para.)
generating, by the positional attention layer, an attention map for each of the context positions based on one or more of the content values associated with the respective context position (The proposed PiCANet simultaneously generates an attention map at each pixel over its context region, pg. 2, left col., first para. the Examiner notes each pixel is context position that has a content value at each pixel) and
a neighborhood of context positions relative to the respective context position (Thus, we enable each pixel (w, h) to “see” the local neighbouring region F¯w,h ∈ RW¯×H¯×C centered at it. pg. 4, left col., first para. The Examiner notes the neighboring region of a pixel is a neighborhood of context positions),
the positional attention layer comprising at least a column- focused attention sublayer that attends to context positions along a column of each respective context position (Next, the ReNet uses another two LSTMs to scan each column of the obtained feature map in both bottom-up and top-down orders, pg. 3, right col., second para. The Examiner notes second top and bottom feature maps in Figure 2a is the column sublayer) and
a row-focused attention sublayer that attends to context positions along a row of each respective context position (Specifically, two LSTMs along each row of F scan the pixels one-by-one from left to right and from right to left, respectively, pg. 3, right col., second para. The Examiner notes first top and bottom feature maps in Fig. 2a is the row sublayer);
Since Guo as primary reference desires a novel end-to-end global-local attention
network (GLANet) (GALNet is a pre-trained fully convolutional neural network, pg. 67202, left col., second para.) is proposed to capture both global and local information (abstract) and Liu as secondary reference discloses convolution operations with attending to global or local context (abstract), then,
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Guo to incorporate the teachings of Liu for the benefit of constructing a network by hierarchically embedding (pg. 2, third para.) and learning of global attention, which can help to learn global contrast better and improve model performance (Liu, pg. 2, right col., first para.)
Regarding claim 4, Modified Guo teaches the computing system of claim 1, Liu teaches wherein the column- focused attention sublayer (The GAP network architecture is shown in Figure 2(a) … Next, the ReNet uses another two LSTMs to scan each column of the obtained feature map in both bottom-up and top-down orders, pg. 3 right col., second para.) and
row-focused attention sublayer (The GAP network architecture is shown in Figure 2(a) … Specifically, two LSTMs along each row of F scan the pixels one-by one from left to right and from right to left, respectively, pg. 3 right col., second para.) are configured to operate in parallel with each other (The first top and bottom feature maps in Fig. 2a is the column sublayer, and both sublayers are parallel with each other).
The same motivation to combine independent claim 1 applies here.
Regarding claim 6, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the column-focused attention sublayer (The GAP network architecture is shown in Figure 2(a) … Next, the ReNet uses another two LSTMs to scan each column of the obtained feature map in both bottom-up and top-down orders, pg. 3, right col., second para.) and the row-focused attention sublayer (he GAP network architecture is shown in Figure 2(a) … Specifically, two LSTMs along each row of F scan the pixels one-by one from left to right and from right to left, respectively, pg. 3, right col., second para.) each are configured to use learned relative positional embeddings for each respective context position (We present three formulations of the PiCANet via embedding the pixel-wise contextual attention mechanism (abstract); PiCANets sample limited context positions by using dilation or adopting local PiCANets, and thus can be applied to feature maps at various scales, pg. 5, right col., second to the last para.).
The same motivation to combine independent claim 1 applies here.
Regarding claim 8, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the output of the positional attention layer is determined based at least in part on combining output from each of the attention sublayers (output of first top and bottom feature maps in Fig. 2a (as row sublayer) is combined with output of second top and bottom feature maps in Fig. 2a (as column sublayer); As a result, we obtain an attentive contextual feature map FGAP, pg. 3, right col., second to the last para.).
The same motivation to combine independent claim 1 applies here.
Regarding claim 9, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the machine-learned model has been trained on a set of labeled training data using supervised learning (we add explicit supervision for the learning of global attention, pg. 2, right col., first para.),
wherein the supervised learning comprises backpropagating a gradient of a loss function through a plurality of parameters (The whole network is trained end-to-end using stochastic gradient descent (SGD) with momentum. For the weight of each loss term in, we empirically set γ6,γ5 ,...,γ1 as 0.5, 0.5, 0.5, 0.8, 0.8, and 1, respectively, without further tuning, pg. 7, first para.).
The same motivation to combine independent claim 1 applies here.
Regarding claim 10, Modified Guo teaches the computer system of claim 1, Guo teaches wherein the input data comprises at least one of image data, video data, sensor data, audio data, or text data (Given an input image, we can extract the feature map X through the backbone network with size H x W x C, pg. 67202, left col., second to the last para.)
Regarding claim 11, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the machine-learned model has been trained to perform image recognition, image classification, image captioning, scene segmentation, object detection, action recognition, action localization, image synthesis, semantic segmentation, panoptic segmentation, or natural language processing (Furthermore, we demonstrate the effectiveness and generalization ability of the PiCANets on semantic segmentation and object detection with improved performance, abstract).
The same motivation to combine independent claim 1 applies here.
Regarding claim 12, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the machine-learned model has been trained on a set of ImageNet training data (The VGG16 network is used as our encoder to utilize its parameters pre-trained on ImageNet, pg. 6, left col., second para.).
The same motivation to combine independent claim 1 applies here.
Regarding claim 13, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the machine-learned model is used as part of backbone processing in a neural network (For object detection, we embed the Pi CANets into the SSD network for experiments. SSD uses the VGG [49] 16-layer network as the backbone, pg. 12, left col., last para.).
The same motivation to combine independent claim 1 applies here.
Regarding claim 14, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the machine-learned model is used to replace convolutions in a neural network (When using AC, we directly replace the vanilla Conv layer of Conv8 2 with an AC module, pg. pg. 12, right col., first para.); the AC (attention convolution) module generate attention over a local neighboring region¯ Fw,h centered at (w,h), pg. 3, left col., last para.; Fig. 2. (f) show detailed operations of AC).
The same motivation to combine independent claim 1 applies here.
Regarding claim 15, Modified Guo teaches the computer system of claim 1, Liu teaches wherein a sequence of two or more instances of the machine-learned model (In this section, we present three forms of the proposed PiCANet. Suppose we have a convolutional (Conv) feature map F ∈ RW×H×C, with W, H, and C denoting its width, height and number of channels, respectively. For each location (w,h) in F, the GAP module generates global attention over the entire feature map F, while the LAP module and the AC module generate attention over a local neighbouring region ¯ Fw,h centered at (w,h), pg. 3, left col., last para.; All the three models are fully differentiable and can be integrated with convolutional neural networks with joint training, abstract. the Examiner notes the three instances are GAP module, LAP module and AC module) are implemented as part of a neural network (The results demonstrate that PiCANets can be used as general neural network modules for dense prediction tasks, pg. 2, left col., third para.).
The same motivation to combine independent claim 1 applies here.
Regarding claim 16, Modified Guo teaches the computer system of claim 1, Liu teaches wherein the sequence of the two or more instances of the machine-learned model (In this section, we present three forms of the proposed PiCANet. Suppose we have a convolutional (Conv) feature map F ∈ RW×H×C, with W, H, and C denoting its width, height and number of channels, respectively. For each location (w,h) in F, the GAP module generates global attention over the entire feature map F, while the LAP module and the AC module generate attention over a local neighbouring region ¯ Fw,h centered at (w,h), pg. 3, left col., last para.; All the three models are fully differentiable and can be integrated with convolutional neural networks with joint training, abstract. the Examiner notes the three instances are GAP module, LAP module and AC module) are arranged consecutively (The three instances are GAP module, LAP module and AC module are arranged consecutively in Fig. 3a) as part of the neural network (The results demonstrate that PiCANets can be used as general neural network modules for dense prediction tasks, pg. 2, left col., third para.).
The same motivation to combine independent claim 1 applies here.
Regarding claim 18, claim 18 is similar to claim 1. It is rejected in the same manner and reasoning applying.
Regarding claim 19, claim 19 is similar to claim 1. It is rejected in the same manner and reasoning applying. Further, Guo teaches one or more non-transitory computer-readable media storing one or both of: instructions that when executed by a computing system cause the computing system to perform operations, the operations comprising (an NVIDIA GTX TITAN GPU, pg. 67205, left col., first para.):
4. Claims 3 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. ("Global-local attention network for aerial scene classification." IEEE Access 7 (2019): 67200-67212). in view of Liu et al. ("PiCANet: Pixel-wise contextual attention learning for accurate saliency detection." IEEE Transactions on Image Processing 29 (2020): 6438-6451, Date of Publication: 23 April 2020) and further in view of Huang et al. ("Dsanet: Dual self-attention network for multivariate time series forecasting." Proceedings of the 28th ACM international conference on information and knowledge management. 2019)
Regarding claim 3, Modified Guo teaches the computing system of claim 1, Huang teaches wherein the global attention operation comprises multiplying the queries, a matrix transpose of the keys with softmax normalization applied to each row, and the values (In the self-attention module following the global temporal convolution, a set of queries, keys, and values are packed together into matrices QG, KG, and VG, obtained by applying projections to the input HG, pg. 2130, right col., last para.); Mathematically, the scaled dot product self-attention computation can be expressed as
PNG
media_image1.png
58
400
media_image1.png
Greyscale
where dk is the dimension of keys, pg. 2131, left col., first para.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Guo to incorporate the teachings of Huang for the benefit of applying self-attention mechanism in time series forecasting with the help of well-designed dual branches architecture which demonstrates accuracy and robustness (Huang, pg. 2130, left col., first para.)
Regarding claim 17, Modified Guo teaches the computer system of claim 1, Huang teaches wherein determining the layer-output comprises summing the one or more output features for each context position generated by the content attention layer and the attention map generated for each context position by the positional attention layer (In the forecasting stage, we first use a dense layer to combine the outputs of two self-attention modules and get the self-attention based prediction ˆXDT+h ∈ RD. The final prediction of DSANet ˆXT+h is then obtained by summing the self attention based prediction ˆXDT+h and the AR prediction ˆXLT+h, pg. 2131, left col., second to the last para.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Guo to incorporate the teachings of Huang for the benefit of applying self-attention mechanism in time series forecasting with the help of well-designed dual branches architecture which demonstrates accuracy and robustness (Huang, pg. 2130, left col., first para.)
5. Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. ("Global-local attention network for aerial scene classification." IEEE Access 7 (2019): 67200-67212) in view of Liu et al. ("PiCANet: Pixel-wise contextual attention learning for accurate saliency detection." IEEE Transactions on Image Processing 29 (2020): 6438-6451, Date of Publication: 23 April 2020) and further in view of Li et al. ("Ftrans: energy-efficient acceleration of transformers using fpga." Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design. 2020, August 10–12, 2020).
Regarding claim 2, Modified Guo teaches the computing system of claim 1, Modified Guo does not explicitly teach the limitations of claim 2.
Li teaches wherein the machine-learned model further comprises an input processing layer that generates a plurality of keys, queries, and values derived from the input data (The input consists of queries and keys of dimension dk, and values of dimension dv, pg. 3, right col., third para.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Guo to incorporate the teachings of Li for the benefit of significantly reducing the model size of NLP (Natural Language Processing) models by up to 16 times (Li, abstract).
6. Claim 5 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. ("Global-local attention network for aerial scene classification." IEEE Access 7 (2019): 67200-67212) in view of Liu et al. ("PiCANet: Pixel-wise contextual attention learning for accurate saliency detection." IEEE Transactions on Image Processing 29 (2020): 6438-6451, Date of Publication: 23 April 2020) and further in view of Luo et al. (US20200257979 filed 04/29/2020)
Regarding claim 5, Modified Guo teaches the computing system of claim 1, Modified Guo does not explicitly teach the limitations of 5.
Luo teaches wherein the positional attention layer comprises the column-focused attention sublayer (A dimension normalization unit 42 configured to normalize a feature map set output by means of a network layer [0260]; the dimension normalization unit 42 is configured to obtain the spatial dimension mean based on at least one feature map by using a height value … of the at least one feature map [0268]. The Examiner notes the height is the column) followed by a batch normalization layer (Moreover, the normalization layer is added behind each layer of neural network to perform the adaptive normalization operation on each layer of feature map [0210]) that is followed by the row-focused attention sublayer (A dimension normalization unit 42 configured to normalize a feature map set output by means of a network layer [0260]; the dimension normalization unit 42 is configured to obtain the spatial dimension mean based on at least one feature map by using … a width value of the at least one feature map [0268]. The Examiner notes the width is the row).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Guo to incorporate the teachings of Luo for the benefit of performing normalization along at least one dimension so that statistics information of each dimension of a normalization operation is covered, thereby ensuring good robustness of statistics in each dimension (Luo, abstract)
Regarding claim 7, Modified Guo teaches the computing system of claim 1, Modified Guo does not explicitly teach the limitations of claim 7.
Luo teaches wherein the positional attention layer comprises the column-focused attention sublayer (A dimension normalization unit 42 configured to normalize a feature map set output by means of a network layer [0260]; the dimension normalization unit 42 is configured to obtain the spatial dimension mean based on at least one feature map by using a height value … of the at least one feature map [0268]. The Examiner notes the height is the column) followed by the batch normalization layer (Moreover, the normalization layer is added behind each layer of neural network to perform the adaptive normalization operation on each layer of feature map [0210]) that is followed by the row-focused attention sublayer (A dimension normalization unit 42 configured to normalize a feature map set output by means of a network layer [0260]; the dimension normalization unit 42 is configured to obtain the spatial dimension mean based on at least one feature map by using … a width value of the at least one feature map [0268]. The Examiner notes the width is the row) that is followed by a second batch normalization layer (Moreover, the normalization layer is added behind each layer of neural network to perform the adaptive normalization operation on each layer of feature map [0210], Fig. 3) that is followed by a time or depth attention sublayer (different normalization operation modes are selected in different network depths due to different visual representations [0203]; at least one normalization layer based on the prediction result [0292]; adjusting parameters of the at least one network layer … based on the prediction result [0209]. The Examiner notes the network layer as depth attention sublayer adjust parameters based on the prediction result from the normalization layer).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Modified Guo to incorporate the teachings of Luo for the benefit of performing normalization along at least one dimension so that statistics information of each dimension of a normalization operation is covered, thereby ensuring good robustness of statistics in each dimension (Luo, abstract)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MORIAM MOSUNMOLA GODO whose telephone number is (571)272-8670. The examiner can normally be reached Monday-Friday 8am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.G./Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights