Detailed Action
This Office Action is in response to the remarks entered on 03/30/2026. Claim 19 has been canceled. New Claim 21 has been added. Claims 1-18 and 20-21 are currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Amended claims were received on 03/30/2026. Claim Objections have been withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 9-10 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Fubuki et al. (Fubuki et al. “Attention Branch Network: Learning of Attention Mechanism for Visual Explanation”, 2019, hereinafter ‘Fubuki’) in view of KARANAM & WU (US 20210090289 A1, hereinafter ‘Karanam’).
Regarding claim 1, Fubuki teaches:
A computing system comprising: ([Fubuki, page 1, right col, 2.2. Attention mechanism, line 1-2] and [Fubuki, page 4, left col, 4. Experiments, line 1 – right col, line 11] implies that the method is implemented using a generic computer)
generating, via a feature extraction network, based on an input image, one or more of a first feature map or a second feature map; ([Fubuki, page 2, Fig 2. (a); left col, 3. Attention Branch Network, line 1 – right col, line 7] The feature extractor receives the input image xi and generates the feature map g(xi))
generating, via a feature importance network, a feature importance vector based on combining the input image and the first feature map; ([Fubuki, page 2, Fig 2. (a); left col, line 10-12] and [page 2, left col, 3. Attention Branch Network, line 1 – right col, line 12] The feature extractor and the attention branch generate attention map
M
(
x
i
)
which represents the important region in image recognition. [Fubuki, Figure 3 and page 3, left col, 3.1. Attention branch, line 10-32] also describes the Input image
X
i
is converted to the feature map
g
(
x
i
)
and combined with
M
(
x
i
)
using convolution operation ⊗ to generate g ⊗ M (i.e., combining) which goes into the Attention Branch)
generating, via an activation function, an attention map based on a weighted sum of the feature importance vector and the first feature map; ([Fubuki, page 3, left col, line 2-9 and line 23-41] discloses that the attention map is generated by multiplying the weighted sum of the K x h x w feature map (the first feature map) by the weight at the last fully-connected layer. K feature maps are generated and combined to generate the attention map. Each feature maps in the K feature map can be interpreted as the 1st feature map, the 2nd feature map, and feature importance vector. In addition, the weight at the last fully-connected layer also can be interpreted as the first feature map)
determining a classification output based on combining the attention map and one or more of the first feature map or the second feature map; and ([Fubuki, page 2, Fig. 2; right col, line 3-7] The perception branch outputs the probability of each class (classification output) by receiving the feature map from the feature extractor and attention map. [page 3, left col, 3.2. Perception branch, line 1 – right col, line 6] discloses combining the attention map and feature maps to generate classification probabilities)
generating a feature visualization image by [Fubuki, page 2, Fig 2. (a); left col, line 10-12] and [page 2, left col, 3. Attention Branch Network, line 1 – right col, line 12] The feature extractor and the attention branch generate attention map M(xi) which represents the important region in image recognition. The attention map provides visual explanation on the basis of response-based visual explanation)
However, Fubuki does not specifically disclose:
a processor; and
a memory coupled to the processor, the memory storing instructions which, when executed by the processor, cause the computing system to perform operations comprising:
generating a feature visualization image by overlaying the attention map onto the input image.
Karanam teaches:
a processor; and a memory coupled to the processor, the memory storing instructions which, when executed by the processor, cause the computing system to perform operations comprising: ([Karanam, 0088] discloses utilizing a processor, memory, and instructions to implement the system)
generating a feature visualization image by overlaying the attention map onto the input image. ([Karanam, 0030] and [0042] collectively disclose the output module overlaying an attention map onto the input image)
It would have been obvious before the effective filing date of the claimed invention to a person
having ordinary skill in the art to apply the method of overlaying the attention map onto the input image of Karanam to improve the feature importance learning system of the present invention. The suggestion and/or motivation for doing so is to assist debugging process and improve reliability of the machine learning model by visualizing the inference process of the machine learning model.
Regarding claim 2, Fubuki teaches:
The computing system of claim 1, wherein the feature extraction network comprises a first neural network including a plurality of convolution layers, wherein the first feature map is obtained from a last layer of the plurality of convolution layers, and wherein the feature importance network comprises a second neural network. ([Fubuki, page 2, Fig. 2, (a) and (b); left col, 3. Attention Branch Network, line 1 – right col, line 21] The feature extractor contains multiple convolution layers and extracts feature maps from an input image. The CAM is constructed after the feature extractor and contains K x 3 x 3 convolution layer, GAP, and fully-connected layer. [Fubuki, page 3, left col, 3.2. Perception branch, line 1-6; page 2, Fig. 2 (c)] discloses the image classification models are VGGNet and ResNet which are neural networks)
Claim 9 is a method claim having similar limitation to the claim 1. Therefore, claim 9 is rejected under the same reason as claim 1.
Claim 10 is a method claim having similar limitation to the claim 2. Therefore, claim 10 is rejected under the same reason as claim 2.
Regarding claim 15, Fubuki in view of Karanam teaches:
At least one non-transitory computer readable medium comprising instructions which, when executed by a computing system, cause the computing system to perform operations comprising ([Fubuki, page 1, right col, 2.2. Attention mechanism, line 1-2] and [Fubuki, page 4, left col, 4. Experiments, line 1 – right col, line 11] implies that the method is implemented using a generic computer. Additionally, [Karanam, 0088] discloses utilizing a processor, memory, and instructions to implement the system)
Claim 15 is a non-transitory computer readable medium claim having similar limitation to the claim 1. Therefore, claim 15 is rejected under the same reason as claim 1.
Claim 16 is a non-transitory computer readable medium claim having similar limitation to the claim 2. Therefore, claim 16 is rejected under the same reason as claim 2.
Claims 3, 5-6, 11, 13, 17 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Fubuki in view of Karanam and further in view of Zhou et al. (US 20230026811 A1, hereinafter ‘Zhou’).
Regarding claim 3, Fubuki teaches:
The computing system of claim 2.
However, Fubuki does not specifically disclose:
wherein the second feature map is obtained from an intermediate layer, other than the last layer, of the plurality of convolution layers.
Zhou teaches:
wherein the second feature map is obtained from an intermediate layer, other than the last layer, of the plurality of convolution layers. ([Zhou, 0091 and 0093] discloses inputting the input image into a shallow feature extractor and generating a plurality of intermediate feature maps. The concatenation module 306 generates a combined feature map by combining the plurality of intermediate feature maps and the last feature map 303M)
It would have been obvious before the effective filing date of the claimed invention to a person
having ordinary skill in the art to apply the method of obtaining and combining intermediate feature maps of Zhou to improve the feature importance learning system of the present invention. The suggestion and/or motivation for doing so is to improve the accuracy of the machine learning model output by averaging out the noise contained in the feature map.
Regarding claim 5, Fubuki teaches:
The computing system of claim 3, wherein combining the attention map and one or more of the first feature map or the second feature map comprises: ([Fubuki, page 3, left col, 3.2. Perception branch, line 6 - right col, Equation (2), line 9] The feature maps
g
c
(
x
i
)
are combined with attention maps
M
(
x
i
)
using a dot-product operation (element-wise multiplication function))
generating an output map by combining, via an attention mechanism, the attention map and one or more of the first feature map or the second feature map; and ([Fubuki, page 3, left col, 3.2. Perception branch, line 6 - right col, Equation (2), line 9] The feature maps
g
c
(
x
i
)
(includes the fist and the second feature map) are combined with attention maps
M
(
x
i
)
using a dot-product operation (element-wise multiplication function) )
applying an activation function to the output map. ([Fubuki, page 3, left col, 3.2. Perception branch, line 1-6; page 2, Fig. 2(c)] and [Fubuki, page 4, left col, line 27-34] collectively discloses that the perception branch receives the converted feature map converted by applying attention map
M
t
x
at specific task t. The perception branch is a neural network classifier which consists of a plurality of activation functions)
Regarding claim 6, Fubuki teaches:
The computing system of claim 5, wherein generating an attention map based on a weighted sum of the feature importance vector and the first feature map comprises: ([Fubuki, page 3, left col, line 2-9 and line 23-41] discloses that the attention map is generated by multiplying the weighted sum of the K x h x w feature map (the first feature map) by the weight (feature importance vector) at the last fully-connected layer. K feature maps are generated and combined to generate the attention map. Each feature maps in the K feature map are interpreted as the 1st feature map and the 2nd feature map. Claim 6 tells that the weights are the feature importance vector)
computing a specific weighted sum
Σ
k
=
1
N
w
k
F
M
k
, wherein weights
w
k
are derived from respective coefficients of the feature importance vector, and
F
M
k
is a k-th channel of the first feature map; and ([Fubuki, page 3, left col, line 2-9 and line 23-41] discloses that the attention map is generated by multiplying the weighted sum of the K x h x w feature map (the first feature map) by the weight (feature importance vector) at the last fully-connected layer. K feature maps are generated and combined to generate the attention map. Each feature maps in the K feature map are interpreted as the 1st feature map and the 2nd feature map. Claim 6 tells that the weights are the feature importance vector)
applying an activation function to a result of the specific weighted sum; and ([Fubuki, page 3, left col, line 2-9 and line 23-41] The aggregated K feature maps are normalized by the sigmoid function)
wherein the attention mechanism comprises an equation
F
O
=
F
L
⊗
(
1
+
A
M
)
, wherein
F
O
is the output map,
F
L
is the one or more of the first feature map or the second feature map,
A
M
is the attention map, and
⊗
denotes an element-wise multiplication function. ([Fubuki, page 3, left col, 3.2. Perception branch, line 6 - right col, Equation (2), line 9] The feature maps
g
c
(
x
i
)
are combined with attention maps
M
(
x
i
)
using a dot-product operation (element-wise multiplication function) )
Claim 11 is a method claim having similar limitation to the claim 3. Therefore, claim 11 is rejected under the same reason as claim 3.
Claim 13 is a method claim having similar limitation to the claim 5. Therefore, claim 13 is rejected under the same reason as claim 5.
Claim 17 is a non-transitory computer readable medium claim having similar limitation to the claim 3. Therefore, claim 17 is rejected under the same reason as claim 3.
Regarding claim 21, Fubuki in view of Karanam and further in view of Zhou teaches:
wherein the activation function comprises a rectified linear unit function. ([Zhou, 0098] The structure includes a rectified linear unit (ReLU) layer 346)
Claims 4, 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Fubuki in view of Karanam and further in view of Yu et al. (Yu et al. “Region Normalization for Image Inpainting”, 2020, hereinafter ‘Yu’).
Regarding claim 4, Fubuki in view of Karanam teaches:
The computing system of claim 2.
However, Fubuki in view of Karanam does not specifically disclose:
wherein combining the input image and the first feature map comprises:
generating an intermediate image by applying one or more of a downsize function or a greyscale function to the input image;
generating an intermediate feature map by applying a normalize function to the first feature map; and
generating a masked image by multiplying, via element-wise multiplication, the intermediate image and the intermediate feature map.
Yu teaches:
wherein combining the input image and the first feature map comprises: ([Yu, page 12736, right col, Figure 3 (b)] The Figure 3 discloses combining the (MaxPool, AvgPool) image and the normalized input feature)
generating an intermediate image by applying one or more of a downsize function or a greyscale function to the input image; ([Yu, page 12736, Figure 3; left col, 3.3 Learnable Region Normalization, line 14 – right col, line 23] discloses generating a feature (image) by applying max-pooling and average-pooling which are downsampling (downsizing) functions)
generating an intermediate feature map by applying a normalize function to the first feature map; and ([Yu, Figure 3; page 12736, right col, line 8-18] discloses generating a normalized feature (intermediate feature map) by normalizing the input feature F)
generating a masked image by multiplying, via element-wise multiplication, the intermediate image and the intermediate feature map. ([Yu, Figure 3; page 12736, right col, line 8-18] The Region normalization result and
γ
are combined using matrix multiplication. The symbol ⊗ denotes matrix multiplication which is an element-wise multiplication)
It would have been obvious before the effective filing date of the claimed invention to a person
having ordinary skill in the art to apply the method of masking the input image by multiplying the image and the intermediate feature map of Yu to improve the feature importance learning system of the present invention. The suggestion and/or motivation for doing so is to improve the accuracy of the machine learning model output by emphasizing relevant features thereby help visualizing the feature importance more effectively.
Claim 12 is a method claim having similar limitation to the claim 4. Therefore, claim 12 is rejected under the same reason as claim 4.
Claim 18 is a non-transitory computer readable medium claim having similar limitation to the claim 4. Therefore, claim 18 is rejected under the same reason as claim 4.
Claims 7-8, 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fubuki in view of Karanam and further in view of Hwang et al. (Hwang et al. “Aircraft Detection using Deep Convolutional Neural Network for Small Unmanned Aircraft Systems”, 2018, hereinafter ‘Hwang’).
Regarding claim 7, Fubuki in view of Karanam teaches:
The computing system of claim 2.
However, Fubuki in view of Karanam does not specifically disclose:
wherein the input image comprises an image of at least a portion of an aircraft or an aircraft component, and wherein the classification output comprises a determination of at least one of an identification of or a state of the aircraft or the aircraft component.
Hwang teaches:
wherein the input image comprises an image of at least a portion of an aircraft or an aircraft component, and wherein the classification output comprises a determination of at least one of an identification of or a state of the aircraft or the aircraft component. ([Hwang, page 4, D. Training the deep convolutional neural network model, line 1-10] and [Hwang, page 5, IV. Aircraft detection test results, line 1 – page 6, line 8] collectively disclose performing aircraft detection task using the neural network model)
It would have been obvious before the effective filing date of the claimed invention to a person
having ordinary skill in the art to apply the method of inputting image with at least a portion of an aircraft or an aircraft component and generating a classification output of Hwang to implement the feature importance learning system of the present invention. The suggestion and/or motivation for doing so is to improve the aircraft industry by automating the aircraft identification process.
Regarding claim 8, Fubuki in view of Karanam and further in view of Hwang teaches:
The computing system of claim 2, wherein at least one of the first neural network or the second neural network is implemented by an artificial intelligence (AI) accelerator. ([Hwang, page 4, D. Training the deep convolutional neural network model, line 1-10] discloses performing the aircraft detection task using a neural network model implemented using a GPU accelerator that accelerates the training process)
Claim 14 is a method claim having similar limitation to the claim 7. Therefore, claim 14 is rejected under the same reason as claim 7.
Claim 20 is a non-transitory computer readable medium claim having similar limitation to the claim 7. Therefore, claim 20 is rejected under the same reason as claim 7.
Response to Arguments
Applicant's arguments filed 03/30/2026 have been fully considered but they are not persuasive.
Arguments: Applicant asserts that FUBUKI discloses that an attention map is generated by multiplying the feature map “by the weight at the last fully connected layer” but does not disclose “generating, via an activation function, an attention map based on a weighted sum of the feature importance vector and the first feature map” as recited in amended claim 1. [Remarks, page 10]
Examiner’s Response: Examiner respectfully disagrees. Fubuki, [page 2, left col, 2nd para, line 1-3] discloses that “ABN is designed to focus on the attention map for visual explanation that represents the important region in image recognition.” It indicates that the attention map visualizes which portion (i.e., feature) of the image is important or not.
Fubuki further explains in [page 3, left col, line 1-9] that the feature map represents the attention location (i.e., important location) for each class. This paragraph indicates that K feature maps are generated and combined (i.e., weighted sum) to generate the attention map, and then the weighted sum is combined with the weight at the last fully-connected layer. Since K feature maps are combined to generate the attention map and each feature maps indicates which portion of the input data is important, feature maps can be interpreted as both ‘feature importance vector’ and ‘feature maps’.
Each feature maps in the K feature map can be interpreted as the 1st feature map, the 2nd feature map, and feature importance vector. Additionally, the weight at the last fully-connected layer also can be interpreted as the first feature map since the weights in the layer form a matrix.
Accordingly, arguments regarding claim 1 are not persuasive. Similarly, arguments regarding claims 9 and 15 are not persuasive. Therefore, arguments regarding dependent claims 2-8, 10-14, 16-18 and 20-21 are not persuasive.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US-20200143204-A1 (This prior art is pertinent because it discloses utilizing feature importance vector (gradient) to generate an output of a machine learning model [0067])
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUN KWON whose telephone number is (571)272-2072. The examiner can normally be reached Monday – Friday 7:30AM – 4:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUN KWON/Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127
Prosecution Insights