Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 3-4, 6-8, 10-11, 13-15, 17-18, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhang et al. (“Adversarial Style Augmentation for Domain Generalization”, arXiv preprint arXiv:2301.12643, 2023), (hereinafter Zhang).
Regarding claim 1, Zhang teaches a system comprising:
processing circuitry; and non-transitory computer readable media (Zhang, “We conduct the classification experiments on the benchmark PACS dataset (Li et al. 2017). There are 9; 991 samples from 7 classes and 4 domains, i.e., Art, Cartoon, Photo, and Sketch… Experiments are conducted under the single source generalization setting, where we train models on samples of one domain and test them on the remaining three domains.”, pg. 5, 1st column, 3rd full paragraph, lines 10-13, Models are implemented using processing circuitry and storage, which is used for storing image and for testing and validating the models.) storing instructions that, when executed by the processing circuitry, configure the processing circuitry to:
execute, by the processing circuitry, an Adversarially Learned Transformations framework (ALT framework) to learn multiple target generalizations from a single source domain, the ALT framework having at least a diversity network, an adversary network, and a classifier (Zhang, “To explore a broader style space beyond that spanned by batch statistics, we propose to generate more diverse statistics perturbation. Similar to (Li et al. 2022), we model the feature statistics (i.e., mean and standard deviation) as Gaussian distributions and utilize the vanilla feature statistics as the means of these Gaussians. In (Li et al. 2022), the standard deviations of these Gaussians are estimated from the current mini-batch. In contrast, inspired by the DG objective (see Equ. (6)), we model these standard deviations as learnable parameters and optimize them via adversarial training, resulting in the Adversarial Style Augmentation (ASA) method. Specifically, by maximizing the task loss w.r.t. the learnable standard deviations, we approach the most sensitive perturbation direction and intensity (i.e., the worst-case domain). Meanwhile, by minimizing the task loss w.r.t. the vanilla task model, we update the model against the worst case domain perturbation. The model is expected to generalize well to tough unseen test domains if it could generalize to the worst-case domain.”, pg. 2, 1st column, 1st full paragraph, “Existing methods typically introduce
μ
t
and
σ
t
with feature statistics within a mini-batch, e.g., by randomly shuffling or interpolating feature statistics across instances (Zhou et al. 2021c; Nuriel, Benaim, and Wolf 2021) or randomly sampling from a Gaussian distribution estimated from the current mini-batch (Li et al. 2022). Such strategies limit the representation capacity of
μ
t
and
σ
t
within a small statistics space. In this paper, we aim to explore larger style spaces by introducing more diverse
μ
t
and
σ
t
.”, pg. 4, 1st column, 1st full paragraph, lines 8-17, An adversarial style augmentation (ASA) method is executed to generalize models to unseen test domains by learning multiple target generalizations from a single source domain. This ASA method includes a diversity network that diversifies model parameters, an adversary network for image augmentation, and a classifier or CNN for task predictions. See CNN’s and SAM’s of Fig. 2. Note the diversity network and adversary network are being interpreted as being within the style augmentation module.);
obtain, by the processing circuitry, an image training dataset having a plurality of input images representing the single source domain (Zhang, “We conduct the classification experiments on the benchmark PACS dataset (Li et al. 2017). There are 9,991 samples from 7 classes and 4 domains, i.e., Art, Cartoon, Photo, and Sketch. We closely follow (Zhou et al. 2021b) to prepare the training and test data, set up the optimization strategy, and conduct model selection. Particularly, we perform existing style augmentation-based DG methods (Nuriel, Benaim, and Wolf 2021; Zhou et al. 2021c; Li et al. 2022; Zhang et al. 2022) under the same setting for fair comparison. Experiments are conducted under the single source generalization setting, where we train models on samples of one domain and test them on the remaining three domains.”, pg. 5, 1st column, 3rd full paragraph, lines 1-13, Models are trained using image samples from four different domains. In specific embodiments, the model is trained on samples from a single domain and tested on the remaining three domains.);
generate, by the processing circuitry utilizing the ALT framework, image perturbations parameterized by the adversarial network as learnable weights of a neural network representing learned image transformations by the adversarial network for the plurality of input images; train, by the processing circuitry, an Artificial Intelligence model (AI model) of the ALT framework to learn generalizations for the single source domain from the plurality of input images of the image training dataset and the learnable weights of the neural network representing the learned image transformations by the adversarial network (Zhang, “To explore a broader style space beyond that spanned by batch statistics, we propose to generate more diverse statistics perturbation. Similar to (Li et al. 2022), we model the feature statistics (i.e., mean and standard deviation) as Gaussian distributions and utilize the vanilla feature statistics as the means of these Gaussians. In (Li et al. 2022), the standard deviations of these Gaussians are estimated from the current mini-batch. In contrast, inspired by the DG objective (see Equ. (6)), we model these standard deviations as learnable parameters and optimize them via adversarial training, resulting in the Adversarial Style Augmentation (ASA) method.”, pg. 2, 2nd column, 1st full paragraph, lines 1-12, Image sample perturbations are parameterized as learnable weights to enable the model to learn image transformation. The model is first trained as a vanilla model on the single source domain to establish baseline performance and then trained with learnable weights for image transformations.);
train, by the processing circuitry, the AI model of the ALT framework to learn the multiple target generalizations from supplemental images generated by the adversarial network; and output, by the processing circuitry, the AI model (Zhang, “Meanwhile, by minimizing the task loss w.r.t. the vanilla task model, we update the model against the worst-case domain perturbation. The model is expected to generalize well to tough unseen test domains if it could generalize to the worst-case domain.”, pg. 2, 1st column, 1st full paragraph, lines 15-19, “Different from most data augmentation methods that introduce augmented samples in the image space (Zhang et al. 2020; Luo et al. 2020), we generate augmented samples in the feature space, which is more computationally efficient”, pg. 3, 1st column, lines 6-10, see Fig. 1, The model is trained to generalize to multiple unseen target domains by learning from supplemental images generated by the adversary network within the style augmentation module. The adversary network augments original sample images by applying perturbations that maximize task loss to identify worst-case domains. The model learns by training on these augmented images until it is sufficiently updated, at which point the trained model is output for validation on unseen domains.).
Regarding claim 3, Zhang teaches the system of claim 1: wherein the AI model is a trained AI model generalized to multiple target domains utilizing the multiple target generalizations learned from the single source domain (Zhang, “Given the training domain(s)
ε
t
r
, DG aims to learn a model f that performs well across a set of unseen but related domains
ε
a
l
l
⊃
ε
t
r
.“, pg. 4, 2nd column, lines 2-4, The model is trained to generalize across multiple unseen domains based on a single source domain.).
Regarding claim 4, Zhang teaches the system of claim 1, wherein the processing circuitry is further configured to: generate, by the processing circuitry utilizing the ALT framework, the supplemental images from the adversarial network based on the image perturbations; and wherein the supplemental images form no part of the image training dataset obtained (Zhang, “Following (Zhou et al. 2021c; Nuriel, Benaim, and Wolf 2021; Li et al. 2022), we implement our style augmentation module (SAM) based on the framework of AdaIN (Huang and Belongie 2017), which performs style transformation by replacing the feature statistics… In this paper, we aim to explore larger style spaces by introducing more diverse
μ
t
and
σ
t
”, pg. 4, 1st column, 1st full paragraph, lines 1-17, During ASA training, supplemental images are generated by applying adversarial perturbations to the original sample images. These supplemental images constitute augmented images that are separate from and used in conjunction with the original samples for model training. See samples of augmented feature statistic as depicted in Fig. 1 for an example.).
Regarding claim 6, Zhang teaches the system of claim 1, wherein the processing circuitry is further configured to: supplement, by the processing circuitry utilizing the ALT framework, the image training dataset with new image transformations derived from images within the image training dataset utilizing data augmentation operations (Zhang, “Zhang, “Following (Zhou et al. 2021c; Nuriel, Benaim, and Wolf 2021; Li et al. 2022), we implement our style augmentation module (SAM) based on the framework of AdaIN (Huang and Belongie 2017), which performs style transformation by replacing the feature statistics”, pg. 4, 1st column, 1st full paragraph, lines 1-17, During ASA training, supplemental images of additional image transformation are generated via augmentation by applying adversarial perturbations to the original sample images.).
Regarding claim 7, Zhang teaches the system of claim 1, wherein the processing circuitry is further configured to: generate, by the processing circuitry utilizing the ALT framework, the learnable weights of the neural network by the adversary network for the plurality of input images of the image training dataset without use of additive noise; and wherein the adversary network generates the learnable weights of the neural network for the plurality of input images of the image training dataset using image transformations of varying size, varying complexity, or varying size and varying complexity (Zhang, “In contrast, inspired by the DG objective (see Equ. (6)), we model these standard deviations as learnable parameters and optimize them via adversarial training, resulting in the Adversarial Style Augmentation (ASA) method. Specifically, by maximizing the task loss w.r.t. the learnable standard deviations, we approach the most sensitive perturbation direction and intensity (i.e., the worst-case domain). Meanwhile, by minimizing the task loss w.r.t. the vanilla task model, we update the model against the worst-case domain perturbation.”, pg. 2, 1st column, 1st full paragraph, lines 8-17, The model generates image transformations by perturbing feature statistics using learnable weights, without requiring additive noise. The feature statistics of original sample images are replaced with this perturbed statistic to produce style transformations. By adversarial optimizing the learnable weights, a set of image transformations, including varying complexity, is generated for model training.).
Claim 8 corresponds to claim 1, reciting a method comprising steps corresponding to the functions of claim 1. Zhang teaches a method (Zhang, see Fig. 2, illustration of the Adversarial Style Augmentation (ASA) method) comprising steps corresponding to the functions of claim 1. As indicated in the analysis of claim 1, Zhang teaches all the limitations according to claim 1. Therefore, claim 8 is rejected for the same reason as claim 1.
Claims 10, 11, 13, and 14 correspond to claims 3, 4, 6, and 7, respectively, reciting a method comprising steps corresponding to the functions of claim 3, 4, 6, and 7, respectively. Zhang teaches a method (see analysis of claim 8) comprising steps corresponding to the functions of claims 3, 4, 6, and 7. As indicated in the analysis of claims 3, 4, 6, and 7, respectively, Zhang teaches all the limitations according to claims 3, 4, 6, and 7, respectively. Therefore, claims 10, 11, 13, and 14 are rejected for the same reasons as claims 3, 4, 6, and 7, respectively.
Claim 15 corresponds to claim 1, reciting a computer-readable storage media storing instruction to execute the functions according to claim 1. Zhang teaches a computer-readable storage media (Zhang, “We conduct the classification experiments on the benchmark PACS dataset (Li et al. 2017). There are 9; 991 samples from 7 classes and 4 domains, i.e., Art, Cartoon, Photo, and Sketch… Experiments are conducted under the single source generalization setting, where we train models on samples of one domain and test them on the remaining three domains.”, pg. 5, 1st column, 3rd full paragraph, lines 10-13, Models are implemented using processing circuitry and storage, which is used for storing image and for testing and validating the models.) storing instruction to execute the functions according to claim 1. As indicated in the analysis of claim 1, Zhang teaches all the limitation according to claim 1. Therefore, claim 15 is rejected for the same reason as claim 1.
Claims 17, 18, and 20 correspond to claims 3, 4, and 6, respectively, reciting a computer-readable storage media storing instruction to execute the functions according to claims 3, 4, and 6, respectively. Zhang teaches a computer-readable storage media (see analysis of claim 15) storing instruction to execute the functions according to claims 3, 4, and 6. As indicated in the analysis of claims 3, 4, and 6, respectively, Zhang teaches all the limitation according to claim 3, 4, and 6, respectively. Therefore, claims 17, 18, and 20 are rejected for the same reasons as claim 3, 4, and 6, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 5, 9, 12, 16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (“Adversarial Style Augmentation for Domain Generalization”, arXiv preprint arXiv:2301.12643, 2023) in view of Moosavi-Dezfooli et al. (“DeepFool: a simple and accurate method to fool deep neural networks”, Proceedings of the IEEE conference on computer vision and pattern recognition. 2016), (hereinafter Moosavi-Dezfooli).
Regarding claim 2, Zhang teaches the system of claim 1. Zhang does not teach wherein the processing circuitry is further configured to: learn, by the processing circuitry utilizing the ALT framework, image transformations from the adversary network determined to result in a failed prediction output by the classifier; and responsive to a determination the image transformations from the adversary network results in the failed prediction output by the classifier, supplement, by the processing circuitry utilizing the ALT framework, the image training dataset with new images using at least the image transformations from the adversary network determined to result in the failed prediction output by the classifier.
However, Moosavi-Dezfooli teaches wherein the processing circuitry is further configured to: learn, by the processing circuitry utilizing the ALT framework, image transformations from the adversary network determined to result in a failed prediction output by the classifier; and responsive to a determination the image transformations from the adversary network results in the failed prediction output by the classifier, supplement, by the processing circuitry utilizing the ALT framework, the image training dataset with new images using at least the image transformations from the adversary network determined to result in the failed prediction output by the classifier (Moosavi-dezfooli, “In this paper, we fill this gap and propose the DeepFool algorithm to efficiently compute perturbations that fool deep networks, and thus reliably quantify the robustness of these classifiers.”, see abstract, “Though deep networks have exhibited very good performance in classification tasks, they have recently been shown to be particularly unstable to adversarial perturbations of the data [18]. In fact, very small and often imperceptible perturbations of the data samples are sufficient to fool state-of-the-art classifiers and result in incorrect classification. (e.g., Figure 1).”, pg. 2574, 1st column, 1st paragraph, lines 4-10, “Assuming now that f is a general binary differentiable classifier, we adopt an iterative procedure to estimate the robustness
∆
x
0
;
f
. Specifically, at each iteration, f is linearized around the current point xi and the minimal perturbation of the linearized classifier is computed… The DeepFool algorithm for binary classifiers is summarized in Algorithm 1 and a geometric illustration of the method is shown in Figure 3.”, pg. 2576, 1st column, lines 8-19, “In this section, we fine-tune the networks of Table 1 on adversarial examples to build more robust classifiers for the MNIST and CIFAR-10 tasks. Specifically, for each network, we performed two experiments: (i) Fine-tuning the network on DeepFool’s adversarial examples, (ii) Fine-tuning the network on the fast gradient sign adversarial examples. We fine-tune the networks by performing 5 additional epochs, with a 50% decreased learning rate only on the perturbed training set.”, pg. 2579, 1st column, last paragraph, lines 1-3 and 2nd column, lines 1-7, A Deepfool algorithm is used to compute a minimal adversarial perturbation for each image in a training set. This minimal perturbation represents the point where the image would result in a failed prediction by the classifier. The process then synthesizes new images corresponding to the minimal perturbation and adds them to the original training set for adversarial training.).
Zhang teaches generalizing to worst-case domains by learning global parameters for feature statistics perturbations across an entire batch (Zhang, pg. 2, 1st columns, 1st and 2nd full paragraphs). Zhang does not teach determining an image transformation that results in a failed prediction for a specific sample and adding the corresponding image for adversarial training. Moosavi-Dezfooli teaches determining a minimal perturbation for each individual image sample that causes a misclassification, and synthesizing new images using that perturbation (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Zhang’s adversarial training by replacing the global batch-level perturbations with the sample-specific minimal perturbation as taught by Moosavi-Dezfooli (Moosavi-Dezfooli, pg. 2576, 1st column, lines 8-19 and pg. 2579, 1st column, last paragraph, lines 1-3 and 2nd column, lines 1-7). The motivation for doing so would have been train the model on minimal adversarial examples, thereby increasing the model’s robustness to adversarial perturbation (as suggested by Moosavi-Dezfooli, “We perform an extensive experimental comparison, and show that 1) our method computes adversarial perturbations more reliably and efficiently than existing methods 2) augmenting training data with adversarial examples significantly increases the robustness to adversarial perturbations.”, pg. 2575, 1st column, 2nd bullet point). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhang with Moosavi-Dezfooli to obtain the invention as specified in claim 2.
Regarding claim 5, Zhang teaches the system of claim 1. Zhang does not teach wherein the processing circuitry is further configured to: generate, by the processing circuitry utilizing the adversarial network of the ALT framework, the supplemental images configured to trigger the AI model having learned the generalizations for the single source domain to output a failed classification prediction; and train, by the processing circuitry, the AI model of the ALT framework to learn increased generalizations to unseen domains which form no part of the image training dataset obtained, wherein the multiple target generalizations form at least a portion of the increased generalizations to the unseen domains learned by the AI model.
However, Moosavi-Dezfooli teaches wherein the processing circuitry is further configured to: generate, by the processing circuitry utilizing the adversarial network of the ALT framework, the supplemental images configured to trigger the AI model having learned the generalizations for the single source domain to output a failed classification prediction; and train, by the processing circuitry, the AI model of the ALT framework to learn increased generalizations to unseen domains which form no part of the image training dataset obtained, wherein the multiple target generalizations form at least a portion of the increased generalizations to the unseen domains learned by the AI model (Moosavi-dezfooli, “In this paper, we fill this gap and propose the DeepFool algorithm to efficiently compute perturbations that fool deep networks, and thus reliably quantify the robustness of these classifiers.”, see abstract, “Though deep networks have exhibited very good performance in classification tasks, they have recently been shown to be particularly unstable to adversarial perturbations of the data [18]. In fact, very small and often imperceptible perturbations of the data samples are sufficient to fool state-of-the-art classifiers and result in incorrect classification. (e.g., Figure 1).”, pg. 2574, 1st column, 1st paragraph, lines 4-10, “Assuming now that f is a general binary differentiable classifier, we adopt an iterative procedure to estimate the robustness
∆
x
0
;
f
. Specifically, at each iteration, f is linearized around the current point xi and the minimal perturbation of the linearized classifier is computed… The DeepFool algorithm for binary classifiers is summarized in Algorithm 1 and a geometric illustration of the method is shown in Figure 3.”, pg. 2576, 1st column, lines 8-19, “In this section, we fine-tune the networks of Table 1 on adversarial examples to build more robust classifiers for the MNIST and CIFAR-10 tasks. Specifically, for each network, we performed two experiments: (i) Fine-tuning the network on DeepFool’s adversarial examples, (ii) Fine-tuning the network on the fast gradient sign adversarial examples. We fine-tune the networks by performing 5 additional epochs, with a 50% decreased learning rate only on the perturbed training set.”, pg. 2579, 1st column, last paragraph, lines 1-3 and 2nd column, lines 1-7, A Deepfool algorithm is used to compute a minimal adversarial perturbation for each image in a training set. This minimal perturbation represents the point where the image would result in a failed prediction by the classifier. The process then synthesizes new images corresponding to the minimal perturbation and adds them to the original training set for adversarial training.).
Zhang teaches generalizing to worst-case domains by learning global parameters for feature statistics perturbations across an entire batch (Zhang, pg. 2, 1st columns, 1st and 2nd full paragraphs). Zhang does not teach generating additional training images which results in a failed prediction for a specific sample. Moosavi-Dezfooli teaches determining a minimal perturbation for each individual image sample that causes a misclassification, and synthesizing new images using that perturbation (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Zhang’s adversarial training by replacing the global batch-level perturbations with the sample-specific minimal perturbation as taught by Moosavi-Dezfooli (Moosavi-Dezfooli, pg. 2576, 1st column, lines 8-19 and pg. 2579, 1st column, last paragraph, lines 1-3 and 2nd column, lines 1-7). The motivation for doing so would have been train the model on minimal adversarial examples, thereby increasing the model’s robustness to adversarial perturbation (as suggested by Moosavi-Dezfooli, “We perform an extensive experimental comparison, and show that 1) our method computes adversarial perturbations more reliably and efficiently than existing methods 2) augmenting training data with adversarial examples significantly increases the robustness to adversarial perturbations.”, pg. 2575, 1st column, 2nd bullet point). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Zhang with Moosavi-Dezfooli to obtain the invention as specified in claim 5.
Claims 9 and 12 corresponds to claims 2 and 5, respectively, reciting a method comprising steps corresponding to the functions of claims 2 and 5, respectively. Zhang in view of Moosavi-Dezfooli teaches a method (see analysis of claim 1) comprising steps corresponding to the functions of claims 2 and 5. As indicated in the analysis of claims 2 and 5, respectively, Zhang in view of Moosavi-Dezfooli teaches all the limitations according to claims 2 and 5, respectively. Therefore, claims 9 and 12 are rejected for the same reasons as claims 2 and 5.
Claims 16 and 19 corresponds to claims 2 and 5, respectively, reciting a computer-readable storage media storing instruction to execute the functions according to claims 2 and 5, respectively. Zhang in view of Moosavi-Dezfooli teaches a computer-readable storage media (see analysis of claim 15) storing instruction to execute the functions according to claims 2 and 5. As indicated in the analysis of claims 2 and 5, respectively, Zhang in view of Moosavi-Dezfooli teaches all the limitation according to claims 2 and 5, respectively. Therefore, claims 16 and 19 are rejected for the same reasons of obviousness as claims 2 and 5, respectively.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONNOR LEVI HANSEN whose telephone number is (703)756-5533. The examiner can normally be reached Monday-Friday 9:00-5:00 (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONNOR L HANSEN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights