DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the Amendment filed on 8/17/2026
Claims 1-4, 6-13, 15-22 are pending. Claims 1, 10, 19 have been amended. Claims 5, 14 have been cancelled.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 2, 6-9, 10, 11, 15-18 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 4, (1, 3), 10, 11, 15-18 (of U.S. Patent No. US 12,120,384 B2 (Application No. 16/856,885). Although the claims at issue are not identical, they are not patentably distinct from each other.
Claim 1 of the instant application is determined to be unpatentable in light of Claim 1 of U.S. Patent No. US 12,120,384 B2 (Appl. No. 16/856,885) based on the reasons below for having substantially similar limitations.
Instant Application Claim 1 (18/913,683)
Claim 1 (US 12,120,384 B2 / 16/856,885)
An apparatus comprising: interface circuitry to receive a media file; machine readable instructions; and at least one processor circuit to be programmed by the instructions to: generate, based on a deepfake classification model, a classification score for the media file; obtain a class output from a final convolution feature map of the deepfake classification model;
An apparatus comprising: a deepfake classification model trainer to train a classification model based on a first portion of a dataset of media files with known classification information, the classification model to output a classification for input media file from a second portion of the dataset of media files with the known classification information; a classification analyzer to: compare the classification of the input media file from the classification model with a known classification of the input media file to determine if a misclassification occurred; and when the input media file is misclassified, determine a reason for the misclassification based on a comparison of first patterns of an explainability map of the misclassified input media file to second patterns of correctly classified explainability maps, the explainability map to identify a part within the input media file responsible for generating the classification of the input media file; a model modifier to modify the classification model based on the reason for the misclassification; and
calculate a gradient value of the class output with respect to the final convolution feature map
pool the gradient value over two spatial axes and a channel axis of the final
convolution feature map to generate a pooled gradient value
weigh a channel of the final convolution feature map with the pooled gradient
value to generate a pooled weighted feature map·
An explainability map generator to: calculate an average gradient value; weigh a channel of a feature map using the average gradient value; and generate the explainability map based on the average weighted feature map along channel dimensions of the feature map.
generate an explainability map based on a pooled weighted feature map along a channel dimension of the feature map; and identify the media file as a real or a deepfake media file based on the classification score and the explainability map.
an explainability map generator to: calculate an average gradient value; weigh a channel of a feature map using the average gradient value; and generate the explainability map based on the average weighted feature map along channel dimensions of the feature map.
Although the claims at issue are not identical, they are not patentably distinct from each other. Both Claim 1 of the instant application and Claim 1 of the reference patent are directed to an apparatus that employs a deepfake classification model and generates an explainability map by weighing a channel of a feature map using a gradient value. The instant application's claim recites a "pooled weighted feature map along a channel dimension," which is merely a broader recitation of the reference patent's "average weighted feature map along channel dimensions." The instant application's recitation of "obtaining a class output from a final convolution feature map" and "generating an explainability map based on a pooled weighted feature map" is fully encompassed by, and would have been obvious over, the reference patent's teaching of calculating an average gradient value, weighing a channel of a feature map using the average gradient value, and generating the explainability map based on the average weighted feature map along channel dimensions. While Claim 1 of instant application discloses additional limitation “calculate a gradient value of the class output with respect to the final convolution feature map pool the gradient value over two spatial axes and a channel axis of the final convolution feature map to generate a pooled gradient value
weigh a channel of the final convolution feature map with the pooled gradient
value to generate a pooled weighted feature map”. It is not identical as Claim 1 of (US 12,120,384 B2 / 16/856,885), however, prior art Selvaraju teaches in Page 3-4, Section 3 that use Grad-CAM technique that computes the gradient of the class score with respect to the final convolutional layer's feature maps, global-average-pools those gradients over the spatial dimensions to produce a per-channel importance weight, and combines those per-channel weights with the feature maps by summing across the channel axis to produce the localization/explainability map. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate gradient-computation, global-average-pooling of gradients over spatial axes and across channels, and weighted per-channel combination of feature maps taught by Selvaraju into modified invention of Claim 1 of (US 12,120,384 B2 / 16/856,885) such that per-channel weight is obtained as the pooled gradient reaction region map is generated as the pooled-gradient-weighted combination of feature maps along the channel dimension. Therefore, Claim 1 of (US 12,120,384 B2 / 16/856,885) with Selvaraju discloses all limitations of instant application Claim 1.
Claim 2 of the instant application is determined to be unpatentable in light of Claim 4 of U.S. Patent No. US 12,120,384 B2 (Appl. No. 16/856,885) based on the reasons below for having substantially similar limitations.
Instant Application Claim 2 (18/913,683)
Claim 4 (US 12,120,384 B2 / 16/856,885)
The apparatus of claim 1, wherein the deepfake classification model is a neural network trained based on a dataset of media files that has known classifications.
The apparatus of claim 1, wherein the classification model is a neural network.
Although the claims are not identical, Claim 2 of the instant application and Claim 4 of the reference patent both recite that the deepfake/classification model is implemented as a neural network. The instant application additionally specifies that the neural network is "trained based on a dataset of media files that has known classifications," which is consistent with and inherently disclosed by the reference patent's teaching of training a classification model based on "a first portion of a dataset of media files with known classification information" in Claim 1. Accordingly, Claim 2 would have been obvious over Claim 4 in view of Claim 1 of the reference patent.
Claims 6, 7, 8, and 9 of the instant application are determined to be unpatentable in light of Claims 1 and 3 of U.S. Patent No. US 12,120,384 B2 (Appl. No. 16/856,885) based on the reasons below for having substantially similar limitations.
Instant Application Claims 6–9 (18/913,683)
Claims 1 and 3 (US 12,120,384 B2 / 16/856,885)
Claim 6: The apparatus of claim 1, wherein the one or more of the at least one processor circuit is to determine whether the media file has been misclassified.
Claim 1: a classification analyzer to: compare the classification of the input media file from the classification model with a known classification to determine if a misclassification occurred.
Claim 7: …is to identify reasons for misclassification.
Claim 1: …determine a reason for the misclassification based on a comparison of first patterns of an explainability map of the misclassified input media file to second patterns of correctly classified explainability maps.
Claim 8: …to i) identify patterns of correctly classified media files, ii) compare explainability map of misclassified media file to correctly classified explainability maps, or iii) prompt a user to input the reasons for the misclassification.
Claim 3: …by transmitting a prompt to a user to identify the reason, a response to the prompt corresponding to the reason for the misclassification.
Claim 9: …to modify the deepfake classification model based on the reasons for the misclassification.
Claim 1: a model modifier to modify the classification model based on the reason for the misclassification.
Although the claims are not identical, Claims 6–9 of the instant application collectively cover the same inventive concept as Claims 1 and 3 of the reference patent: (1) determining whether a misclassification occurred, (2) identifying reasons for misclassification by comparing explainability maps or prompting a user, and (3) modifying the classification model based on the identified reasons. The instant application's dependent Claims 6–9 merely present these limitations in a serial, cascading dependent claim structure, while the reference patent consolidates them in Claim 1. The instant application's claims would have been obvious over the reference patent's Claims 1 and 3.
Claims 10, 11, 15-18, they recite limitations similar in scope to the limitations of Claims 1, 2, 6-9, but as a non-transitory computer-readable medium, which determined to be obvious in light of claim 1, 2, 6-9 of US 12,120,384 B2 (Appl. No. 16/856,885) which recite limitations similar in scope to the limitations of 5, 6, 15-18 of US 12,120,384 B2 (Appl. No. 16/856,885) based on same reason described above for having similar limitations as described above for Claims 1, 2, 6-9.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kumagai et al. (US 2022/0245927 A1, hereinafter "Kumagai") in view of Selvaraju et al. ("Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization," 2017, hereinafter "Selvaraju"), and further in view of Alsallakh et al. (US 10,984,054 B2, hereinafter "Alsallakh").
Regarding Claim 1, Kumagai teaches: an apparatus comprising (Kumagai, Paragraph [0036], "the identification result explanation device 100 includes a CPU (Central Processing Unit) 41, a ROM (Read Only Memory) 42, a RAM (Random Access Memory) 13, a storage 44, an input unit 45, a display unit 46, and a communication interface (I/F) 47"); interface circuitry to receive a media file (Kumagai, Paragraph [0041], "The communication interface 47 is an interface for communicating with another device such as a terminal and uses, e.g., standards of the Ethernet (registered trademark), an FDDI, the Wi-Fi (registered trademark), or the like"); machine readable instructions (Kumagai, Paragraph [0037], "in the ROM 42 or the storage 44, an identification result explanation program is stored"); (Kumagai, Paragraph [0042], "Each of the functional configurations is implemented by the CPU 41 by reading the identification result explanation program stored in the ROM 42 or the storage 44, deploying the identification result explanation program into the RAM 43, and executing the identification result explanation program"); and at least one processor circuit to be programmed by the instructions to (Kumagai, Paragraph [0037], "The CPU 41 is a central processing unit which executes various programs and controls each of the units. In other words, the CPU 41 reads a program from the ROM 42 or the storage 44 and executes the program by using the RAM 43 as a workspace"): generate, based on a deepfake classification model, a classification score for the media file (Kumagai, Paragraph [0043], "The trained CNN 2 is a CNN having any given network structure for identifying an object present in an image. The trained CNN 2 stores, e.g., Deep Residual Network proposed in NPL 2 as a trained CNN model. The CNN model mentioned herein is a model having a network structure of a neural network which repeatedly performs a combination of convolutional processing and pooling processing and trained parameter information"); (Kumagai, Paragraph [0067], "a weight wc n is calculated based on a value Ai, j n of the position of each of the feature maps, on an output value yc based on a probability of the identification result c output from the CNN"); it is noted that yc is the probability-based classification score. Kumagai does not explicitly disclose that the classification model is a "deepfake" classification model; however, the structural architecture of a CNN used for deepfake detection is identical to that of any other CNN binary classifier, and training a CNN to distinguish "real" from "deepfake" media is merely applying the known CNN classification framework to the well-recognized problem of deepfake detection. Therefore, it would have been obvious to one of ordinary skill in the art to train the CNN of Kumagai to perform deepfake classification); obtain a class output from a final convolution feature map of the deepfake classification model (Kumagai, Paragraph [0056], "the feature map calculation unit 11 calculates, when the input image 4 is input to the CNN, each of the feature maps output individually from the plurality of filters used in the specified layers of the CNN"); (Kumagai, Paragraph [0058], "The feature map An(x) <read on final convolution feature map> mentioned herein is a matrix, and assumedly has a size of h×w. For example, in a 4-layered res5c in Residual Network-152 in Reference 1, the number N of filters is 2,048, and 2,048 feature maps An(x) can be obtained for each one image"); (Kumagai, Paragraph [0067], "an output value yc based on a probability of the identification result c output from the CNN"); it is noted the class output yc is obtained from processing the final convolution feature map); [[calculate a gradient value of the class output with respect to the final convolution feature map;]] [[pool the gradient value over two spatial axes and a channel axis of the final convolution feature map to generate a pooled gradient value;]] [[weigh a channel of the final convolution feature map with the pooled gradient value to generate a pooled weighted feature map;]] generate an explainability map based on the pooled weighted feature map along a channel dimension of the feature map (Kumagai, Paragraph [0076], "the output unit 17 calculates, for the input image x, the input image reaction region map <read on explainability map> in accordance with Expression (2) shown below by using the feature maps Ak n(x) and the weights wc n"); it is noted that Expression (2) weighs each channel n of the feature map with the weight wc n and sums along the channel dimension, producing the explainability map (reaction region map); and [[identify the media file as a real or a deepfake media file based on the classification score and the explainability map.]]
But Kumagai does not explicitly disclose (i) calculate a gradient value of the class output with respect to the final convolution feature map, (ii) pool the gradient value over two spatial axes and a channel axis of the final convolution feature map to generate a pooled gradient value, (iii) weigh a channel of the final convolution feature map with the pooled gradient value to generate a pooled weighted feature map, and (iv) identify the media file as a real or a deepfake media file based on the classification score and the explainability map.
Selvaraju teaches calculate a gradient value of the class output with respect to the final convolution feature map (Selvaraju, Section 3 (Approach), Page 3, "in order to obtain the class-discriminative localization map Grad-CAM Lc Grad-CAM ∈ Ru×v of width u and height v for any class c, we first compute the gradient of the score for class c, yc (before the softmax), with respect to feature maps Ak of a convolutional layer, i.e. ∂yc/∂Ak"); (Selvaraju, Section 3 (Approach), Page 3, "Grad-CAM uses the gradient information flowing into the last convolutional layer of the CNN to understand the importance of each neuron for a decision of interest"); pool the gradient value over two spatial axes and a channel axis of the final convolution feature map to generate a pooled gradient value (Selvaraju, Section 3 (Approach), Page 4, Equation (1), "αc k = (1/Z) Σi Σj ∂yc/∂Ak ij", together with the accompanying text "These gradients flowing back are global-average-pooled to obtain the neuron importance weights αc k"); it is noted that the pooling operation Σi Σj is performed over the two spatial axes i and j of the final convolutional feature map, and is performed for each feature map k across the channel axis to produce per-channel pooled gradient values αc k spanning the channel axis; (Selvaraju, Section 3 (Approach), Page 4, "This weight αc k represents a partial linearization of the deep network downstream from A, and captures the 'importance' of feature map k for a target class c"); weigh a channel of the final convolution feature map with the pooled gradient value to generate a pooled weighted feature map (Selvaraju, Section 3 (Approach), Page 4, Equation (2), "Lc Grad-CAM = ReLU (Σk αc k Ak)", accompanied by "We perform a weighted combination of forward activation maps, and follow it by a ReLU to obtain"); it is noted that each channel Ak of the final convolution feature map is weighted by the pooled gradient αc k, yielding a pooled weighted feature map along the channel dimension.
Selvaraju and Kumagai are analogous since both deal with generating visual explanation/localization maps for CNN-based image classifiers by combining feature maps with per-feature-map weights. Kumagai provides an identification result explanation device that computes feature maps at a specified convolutional layer and generates a reaction region map by combining feature maps with per-channel weights. Selvaraju provides the Grad-CAM technique that computes the gradient of the class score with respect to the final convolutional layer's feature maps, global-average-pools those gradients over the spatial dimensions to produce a per-channel importance weight, and combines those per-channel weights with the feature maps by summing across the channel axis to produce the localization/explainability map. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Selvaraju's gradient-computation, global-average-pooling of gradients over spatial axes and across channels, and weighted per-channel combination of feature maps into Kumagai's identification result explanation device such that Kumagai's per-channel weight is obtained as the pooled gradient and Kumagai's reaction region map is generated as the pooled-gradient-weighted combination of feature maps along the channel dimension. The motivation to combine is expressly provided by Kumagai itself in Paragraph [0066], which invites the use of Selvaraju's Expression (1) as an alternative weight computation; the further motivation, discussed by Selvaraju in Section 3 (Page 3), is that "Grad-CAM uses the gradient information flowing into the last convolutional layer of the CNN to understand the importance of each neuron for a decision of interest," producing a class-discriminative localization map that faithfully reflects the model's decision without requiring architectural changes or re-training.
Alsallakh teaches identify the media file as a real or a deepfake media file based on the classification score and the explainability map (Alsallakh, Column 17, Line 52-54, "a border color indicates whether the predicted class label for a sample is top-1 correct, top-5 correct, or otherwise"); it is noted the predicted class label is derived from the probability distribution which reads on the classification score; (Alsallakh, Column 17, Line 55-60, "if a user selects a particular neuron via the response map, the sample viewer 250 shows saliency maps <read on explainability maps> of the sample images, which highlight image features and/or portions of the image that the selected neuron responds to, in order to analyze the role of the selected neuron in the image classification model 30"); (Alsallakh, Column 7, Line 45-52, "for each sample image, the received output includes a final output of output layer 114 of the image classification model comprises a probability distribution across each possible class label. In at least one embodiment, for each sample image, the received output comprises a predicted class label for the respective image, corresponding to the most probable class label according to the final output of output layer 114"). Alsallakh does not literally use the words "real" or "deepfake"; however, as noted above, applying such a binary-class CNN framework to distinguish real media from deepfake media is a well-recognized straightforward application of the known CNN classification framework to the well-recognized problem of deepfake detection.
Alsallakh and Kumagai are analogous since both deal with CNN-based image classification models and the visualization/understanding of classification outputs. Kumagai provides an identification result explanation framework producing a reaction region map (explainability map) alongside a probability-based identification result. Alsallakh provides a visual analytics system that jointly uses the predicted class label (derived from the classification score) and the saliency maps (explainability maps) to identify how the model classifies a sample. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Alsallakh's joint use of the predicted class label and the saliency map into the modified Kumagai/Selvaraju device such that the classified media file is identified as belonging to one of two classes (real vs. deepfake) based on both the CNN's classification score and the pooled-gradient explainability map. The motivation is to provide the end-user with an actionable, trustworthy classification decision that is supported by an interpretable visual explanation of the CNN's reasoning, as discussed by Alsallakh in Column 4, Lines 25-30.
Regarding Claim 2, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention in Claim 1.
The combination further teaches wherein the deepfake classification model is a neural network trained based on a dataset of media files that has known classifications (Kumagai, Paragraph [0043], "The trained CNN 2 is a CNN having any given network structure for identifying an object present in an image. The trained CNN 2 stores, e.g., Deep Residual Network proposed in NPL 2 as a trained CNN model. The CNN model mentioned herein is a model having a network structure of a neural network which repeatedly performs a combination of convolutional processing and pooling processing and trained parameter information. The trained CNN 2 has been trained by using an image set corresponding to a set of images. To each of the images in an image set 31, a ground truth label used during CNN training is given. The ground truth label is, e.g., a character string representing the object observed in the image"; it is noted that the image set 31 with ground truth labels is the dataset of media files with known classifications).
Regarding Claim 3, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention in Claim 1.
The combination further teaches wherein the classification score is a probability score corresponding to whether the media file is real or deepfake (Kumagai, Paragraph [0067], "a weight wc n is calculated based on a value Ai, j n of the position of each of the feature maps, on an output value yc based on a probability of the identification result c output from the CNN, and on a value z determining the size of the feature map"; it is noted y^c is the probability-based classification score, the structural architecture of a CNN used for deepfake detection is identical to any other CNN binary classifier. Training such a CNN to distinguish "real" from "deepfake" media is merely applying the known CNN classification framework to the well-recognized problem of deepfake detection).
Alsallakh further teaches classification score is a probability score corresponding to whether the media file is real or deepfake (Alsallakh, Column 7, Line 45-50, "Visual Analytics Program" section, "the received output includes a final output of output layer 114 of the image classification model comprises a probability distribution across each possible class label. In at least one embodiment, for each sample image, the received output comprises a predicted class label for the respective image, corresponding to the most probable class label according to the final output of output layer 114"). The "real or deepfake" aspect is the same obviousness argument as for Claim 1 (binary application of the probability score to real vs. deepfake classification).
Regarding Claim 4, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention in Claim 1.
The combination further teaches wherein the explainability map identifies significant parts within the media file responsible for generating the classification score (Kumagai, Paragraph [0047], "The explanation information 6 includes a pair of the input image 4 and an input image reaction region map indicating a reaction region of the input image 4"; [0048], "the input image reaction region map and the stored image reaction region maps are heat maps in which the numerical values are represented by density variations <read on significant parts >").
Regarding Claim 6, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention in Claim 1.
The combination further teaches wherein the one or more of the at least one processor circuit is to determine whether the media file has been misclassified (Alsallakh, Column 10, Line 29-35, "The confusion matrix 230 advantageously provides a detailed visualization of classifications and misclassification of the sample images which enables the user to easily identify frequent confusion between certain classes"; Column 11, Line 5-11, "it can be observed that a majority of cells 408 corresponding to misclassification occur near the diagonal. The ordering reveals similarity groups among the classes"; it is noted the confusion matrix explicitly detects misclassification by comparing the predicted class label (derived from the classification score) to the actual (ground-truth) class label).
Alsallakh and Kumagai are analogous since both deal with CNN-based image classification and the use of classification scores and visual maps to understand classification results. Kumagai provides an identification result explanation framework with Grad-CAM-like feature map weighting and reaction region map (explainability map) generation. Alsallakh provides a visual analytics system for understanding classification errors (misclassifications), saliency maps. Therefore, it would have been obvious to one of ordinary skill in the art to incorporate Alsallakh's misclassification detection (confusion matrix) into modified Kumagai's identification result explanation device such that the device determines whether the media file has been misclassified. The motivation is that once an explainability map is generated (as taught by Kumagai), it is a natural and complementary step to determine whether the classification was correct or not, as discussed by Alsallakh in the "Visual Analytics Program" section: the program enables "identifying various sources of classification error."
Regarding Claim 7, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention in Claim 6.
The combination further teaches wherein the one or more of the at least one processor circuit is to identify reasons for misclassification (Alsallakh, Column 16, Line 10-20, "The sample viewer 250 provides several possibilities to explore the samples along with saliency maps of their image features. Understanding the image features that a neuron responds to is important to understand how each class is being detected by the convolution layers of the image classification model 30 and why certain sample images from the class are misclassified"; Column 19, Line 47-52, “the visual anaIytics program 28 also helps in detecting mislabeled samples, such as an image of a lion labeled as monkey, by inspecting misclassified samples having very high prediction probability and very low probability assigned to the ground truth”).
As explained in rejection of claim 1, the obviousness for combining of dealing misclassification of Alsallakh into Kumagai is provided above.
Regarding Claim 8, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention in Claim 7.
The combination further teaches wherein to identify reasons for the misclassification includes the instructions to cause one or more of the at least one processor circuit to i) identify patterns of correctly classified media files, (Alsallakh, Column 17, Line 55-60, "if a user selects a particular neuron via the response map, the sample viewer 250 shows saliency maps of the sample images, which highlight image features and/or portions of the image that the selected neuron responds to, in order to analyze the role of the selected neuron in the image classification model 30"; it is noted the saliency maps of correctly classified images reveal the common feature patterns used for correct classification) ii) compare explainability map of misclassified media file to correctly classified explainability maps [[ or iii) prompt a user to input the reasons for the misclassification ]]; ( Alsallakh, Column 13, Line 45-52, "The response map 235 provides overview of the neuron responses at a selected neural network layer of the image classification model 30 to all sample images in the dataset. The response map 235 advantageously enables a user to identify whether classes in the same group activate a similar set of feature detectors and/or neurons of the convolutional layers of the model"; Column 10, Line 27-33, "The confusion matrix 230 advantageously provides a detailed visualization of classifications and misclassification of the sample images which enables the user to easily identify frequent confusion between certain classes").
As explained in rejection of claim 1, the obviousness for combining of dealing misclassification of Alsallakh into Kumagai is provided above.
Regarding Claim 9, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention in Claim 7.
The combination further teaches wherein the one or more of the at least one processor circuit is to modify the deepfake classification model based on the reasons for the misclassification (Alsallakh, Column 18, Line 49-55, "the image classification model 30 is modified to include additional classification targets 130 during training. The additional classification targets 130 are included only during the training process as separate branches after the first convolutional layer 102, after the second convolutional layer 104, after the third convolutional layer 106, and after the fourth convolutional layer 108"; Column 18, Line 44-47, "This information can be used to adapt a training process of the CNNs to be hierarchy-aware, thereby significantly improving the accuracy and accelerating the training convergence"; it is noted the model modification is based on the reasons for misclassification identified through analysis of classification scores and saliency/response maps (analogous to explainability maps).
As explained in rejection of claim 1, the obviousness for combining of dealing misclassification of Alsallakh into Kumagai is provided above.
Regarding Claim 10, it recites limitations similar in scope to the limitations of claim 1 and the combination of Kumagai, Selvaraju, and Alsallakh teaches all the limitations as of Claim 1. And Kumagai discloses these features can be implemented on a computer readable storage medium (Kumagai, Paragraph [0037], “The CPU 41 is a central processing unit which executes various programs and controls each of the units. In other words, the CPU 41 reads a program from the ROM 42 or the storage 44 and executes the program by using the RAM 43 as a workspace. The CPU 41 performs control of each of the components mentioned above and various arithmetic processing based on the program stored in the ROM 42 or the storage 44”).
Regarding Claim 11, it recites limitations similar in scope to the limitations of Claim 2 and therefore is rejected under the same rationale.
Regarding Claim 12, it recites limitations similar in scope to the limitations of Claim 3 and therefore is rejected under the same rationale.
Regarding Claim 13, it recites limitations similar in scope to the limitations of Claim 4 and therefore is rejected under the same rationale.
Regarding Claim 15, it recites limitations similar in scope to the limitations of Claim 6 and therefore is rejected under the same rationale.
Regarding Claim 16, it recites limitations similar in scope to the limitations of Claim 7 and therefore is rejected under the same rationale.
Regarding Claim 17, it recites limitations similar in scope to the limitations of Claim 8 and therefore is rejected under the same rationale.
Regarding Claim 18, it recites limitations similar in scope to the limitations of Claim 9 and therefore is rejected under the same rationale.
Regarding Claim 19, it recites limitations similar in scope to the limitations of claim 1, but in a system. As shown in the rejection, the combination of Kumagai, Selvaraju, and Alsallakh disclose the limitations of claims 1. Additionally, Kumagai discloses an system that maps to Fig. 1, Fig. 2 and Paragraph [0036]-[0037] (Kumagai, Fig. 1, Fig. 2, Paragraph [0036]-[0037], “the identification result explanation device 100 includes a CPU (Central Processing Unit 41” “The CPU 41 is a central processing unit which executes various programs and controls each of the units”) Thus, Claim 19 is met by Caskey according to the mapping presented in the rejection of claims 1, given the apparatus corresponds to the system.
Regarding Claim 20, it recites limitations similar in scope to the limitations of Claim 2 and therefore is rejected under the same rationale.
Regarding Claim 21, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention of Claim 1.
The combination further teaches wherein the deepfake classification model is to distinguish between a real media file and a deepfake media file based on one or more mesoscopic properties of the media file (Kumagai, Paragraph [0058], "For example, in a 4-layered res5c in Residual Network-152 in Reference 1, the number N of filters is 2,048, and 2,048 feature maps An(x) can be obtained for each one image. A size of each of the feature maps An(x) is 7×7"); it is noted that the feature maps computed at the intermediate/deep convolutional layer of the CNN (e.g., res5c) capture image properties at a mid-level of abstraction between raw pixel intensities and fully abstracted class semantics, which reads on "mesoscopic properties" of the media file; (Selvaraju, Section 3 (Approach), Page 3, "A number of previous works have asserted that deeper representations in a CNN capture higher-level visual constructs. Furthermore, convolutional features naturally retain spatial information which is lost in fully-connected layers, so we can expect the last convolutional layers to have the best compromise between high-level semantics and detailed spatial information. The neurons in these layers look for semantic class-specific information in the image (say object parts)"); it is noted the last convolutional layer's features that balance high-level semantics with detailed spatial information are properties at an intermediate ("mesoscopic") scale of the media file that the CNN uses to make its classification decision.
As explained in rejection of claim 1, the obviousness for combining of adapting the combined system to distinguish real from deepfake media of Selvaraju into Kumagai is provided above.
Regarding Claim 22, the combination of Kumagai, Selvaraju, and Alsallakh teaches the invention of Claim 1.
The combination further teaches wherein to pool the gradient value, the at least one processor circuit is to average the gradient value over the two spatial axes and the channel axis of the final convolution feature map (Selvaraju, Section 3 (Approach), Page 4, Equation (1), "αc k = (1/Z) Σi Σj ∂yc/∂Ak ij"); (Selvaraju, Section 3 (Approach), Page 4, "These gradients flowing back are global-average-pooled to obtain the neuron importance weights αc k"); (Selvaraju, Section 3 (Approach), Page 4, "global average pooling" annotation on Equation (1)); it is noted that the operation (1/Z) Σi Σj is an averaging (global average pooling) of the gradient value ∂yc/∂Ak ij over the two spatial axes i and j of the final convolutional feature map, and that this averaging is performed for each channel k across the channel axis to obtain the per-channel pooled gradient values αc k).
As explained in rejection of claim 1, the obviousness for combining of well-defined, differentiable procedure for computing of Selvaraju into Kumagai is provided above.
Response to Arguments
Applicant noted the rejection of Nonstatutory Double Patenting in the previous office action has typo in the patent number, it stated as US 12,120,394 B2, it should have been US 12,120,384 B2 is persuasive.
In response to the argument, the typo of the patent number has been corrected in this office action along with the modified Nonstatutory Double Patenting based on amended claims. Since no Terminal Disclaimer has been filed at the time of this office action posted. The Nonstatutory Double Patenting rejection maintained the same.
Applicant’s arguments with respect to claim 1, 10, 19, filed on 8/17/2026, with respect to rejection under 35 USC § 103 have been considered but are moot in view of the new ground(s) of rejection. it has now been taught by the combination of Kumagai, Selvaraju, and Alsallakh.
In regard to Claims 2-4, 6-9, 11-18, 20, they directly/indirectly depends on independent Claim 1, 10, 19 respectively. Applicant does not argue anything other than the independent claim 1, 10, 19. The limitations in those claims in conjunction with combination previously established as explained.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YUJANG TSWEI whose telephone number is (571)272-6669. The examiner can normally be reached 8:30am-5:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YuJang Tswei/Primary Examiner, Art Unit 2614
