DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2,4-9,11-12,14-16 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over US Pre-Grant Patent 2019/0354817 (Schlens et al; Schlens) in view of “INSTANCE ADAPTIVE ADVERSARIAL TRAINING: IMPROVED ACCURACY TRADEOFFS IN NEURAL NETS,” Balaji et al; Balaji, arXiv,2019
Regarding claim 1 and analogous claims 8 and 15:
Schlens teaches:
1. A computer-implemented method comprising: defining, by one or more processors, a plurality of augmentations for a dataset for training a learning model,
(Schlens, ¶0020)
“For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image [i.e. A computer-implemented method comprising: defining, by one or more processors, a plurality of augmentations for a dataset for training a learning model,”.”
2. wherein the plurality of augmentations comprise a set of parameterized, random transformations,
(Schlens, ¶0034)
“Thus, in some implementations, the controller model 10 can select a series of operations and the characteristics for each operation [i.e. wherein the plurality of augmentations comprise a set of parameterized,]. As one example, the output of the controller model 10 can be represented as: {(Operation O.sub.1, overall operation probability p.sub.1.sup.o, applied only to bounding shape with probability p.sub.1.sup.b, magnitude m.sub.1), (Operation O.sub.2, overall operation probability applied only to bounding shape with probability p.sub.2.sup.b, magnitude m.sub.2), . . . , (Operation O.sub.N, overall operation probability p.sub.N.sup.o, applied only to bounding shape with probability p.sub.N.sup.b, magnitude m.sub.N)} [i.e. , random transformations,] .”
3. wherein the set of parameterized transformations are sampled randomly from ranges of possible values to ensure that an augmented data sample is novel and satisfies a natural data distribution associated with the dataset,
(Schlens, ¶0055)
“At each iteration, one or more training images 12 can be augmented according to the series of augmentation operations 14 selected by the controller model 10 at the current iteration, thereby generating one or more augmented images 16 [i.e. and satisfies a natural data distribution associated with the dataset,]. Next, a machine-learned object detection model 18 can be trained using the training data including the augmented images 16 generated at the current iteration [i.e. wherein the set of parameterized transformations are sampled randomly from ranges of possible values to ensure that an augmented data sample is novel].”
4. wherein the plurality of augmentations include applying image rotation, applying image contrast, and adding random noise;
(Schlens, ¶0037)
“In some implementations, the set of available augmentation operations can include one or more color operations…[i.e. wherein the plurality of augmentations include…]”
(Schlens, ¶0051)
“A rotate operation that rotates the image portion (e.g., including the bounding box) by magnitude degrees; [i.e. applying image rotation,]”
(Schlens, ¶0043)
“A contrast operation that controls a contrast of the image portion (e.g., a magnitude=0 gives a gray image, whereas a magnitude=1 gives the original image); [i.e. applying image contrast,]”
(Schlens, ¶0047)
“A cutout operation that sets a random square patch of side-length magnitude pixels to gray [i.e. and adding random noise;]”
5. tuning, by one or more processors, the ranges as a regular hyperparameter;
(Schlens, ¶0054)
“For example, the set of discrete and operation-specific available magnitudes can be user-selected hyperparameters [i.e. tuning, by one or more processors, the ranges as a regular hyperparameter;].”
6. applying, by one or more processors, the plurality of augmentations to the dataset;
(Schlens, ¶0054)
“In some implementations, the controller can select the respective augmentation magnitude for at least one of the augmentation operations from a respective set of discrete and operation-specific available magnitudes”
7. training, by one or more processors, the learning model with the augmented dataset;
(Schlens, ¶0055)
“Next, a machine-learned object detection model 18 can be trained using the training data including the augmented images 16 generated at the current iteration.”
8. applying, by one or more processors, the adjusted plurality of augmentations to the dataset;
(Schlens, ¶0055)
“At each iteration, one or more training images 12 can be augmented according to the series of augmentation operations 14 selected by the controller model 10 at the current iteration, thereby generating one or more augmented images 16.”
9. and training, by one or more processors, the learning model with the plurality of adjusted augmentations applied to the dataset, comprising;
(Schlens, ¶0056)
“For example, the controller model 10 can be a recurrent neural network and the reward function can be backpropagated through the recurrent neural network to train the network. In such fashion, the controller model 10 can learn over time to generate augmentation strategies 14 which result in augmented training data 16 which teaches the machine-learned model 18 to perform at an increased performance level.”
Schlens does not explicitly teach; Balaji teaches:
1. measuring, by one or more processors, a respective hardness of one or more data samples in the dataset by running inference on the dataset at an end of each epoch while training the learning model,
(Balaji, pg. 3, Sect. 3, ¶2)
“The proposed algorithm is distinctive in that it uses a different ei for each image xi . Ideally, we would choose each ei to be as large as possible without finding images of a different class within the ei-ball around xi [i.e. measuring, by one or more processors, a respective hardness of one or more data samples in the dataset by running inference on the dataset]. Since we have no a-priori knowledge of what this radius is, we use a simple heuristic to update ei after each epoch. After crafting a perturbation for xi, we check if the perturbed image was a successful adversarial example [i.e. at an end of each epoch while training the learning model,].”
2. wherein hardness of a data sample is a risk of losing a learnable signal responsive to a strength of an augmentation,
(Balaji, pg. 1)
“Our method is inspired by the observation that the constraints enforced by adversarial training are infeasible; for commonly used values of e, it is not possible to achieve label consistency within an -ball of each input image because the balls around images of different classes overlap. This is illustrated on the left of Figure 1, which shows that the e-ball around a “bird” (from the CIFAR-10 training set) contains images of class “deer” (that do not appear in the training set) [i.e. wherein hardness of a data sample is a risk of losing a learnable signal responsive to a strength of an augmentation,].”
Examiner notes that under BRI, IAAT motivates per-sample radii by explaining that for some images the robustness constraint is infeasible at a given epsilon because class regions overlap; it also states that epsilon can be “too large for some images,” causing accuracy to lower. This is construed as “a risk of losing a learnable signal responsive to a strength of an augmentation.”
3. wherein the strength of the augmentation is a single scalar parameter that defines an amount of deviation of augmented data from the dataset;
(Balaji, pg. 3, Algorithm 1)
“Require: PGDk(x, y, e) : Function to generate PGD-k adversarial samples with e norm-bound”
Examiner notes that PGDk(x, y, e), with e norm-bound states that e is the single scalar controlling deviation.
4. adjusting, by one or more processors, the plurality of augmentations for the one or more data samples based on a respective measured hardness of the one or more data samples;
(Balaji, pg. 4, Algorithm 2)
After crafting a perturbation for xi, we check if the perturbed image was a successful adversarial example. If PGD succeeded in finding an image with a differe
PNG
media_image1.png
268
414
media_image1.png
Greyscale
Examiner notes that Algorithm 2 makes a per-sample measurement explicit by checking model’s prediction on a perturbed sample and setting e accordingly.
5. increasing, by one or more processors, a respective strength of the plurality of augmentations to increase respective hardness of each data sample in the dataset
(Balaji, pg. 3, Sect. 3, Algorithm 2)
“After crafting a perturbation for xi , we check if the perturbed image was a successful adversarial example. If PGD succeeded in finding an image with a different class label, then i is too big, so we replace ei ← ei − γ. If PGD failed, then we set ei ← ei + γ.”
PNG
media_image1.png
268
414
media_image1.png
Greyscale
One of ordinary skill in the art, at the time the invention was filed, would have been motivated to modify Schlens with Balaji. A POSITA would have been motivated to combine Shlens with Balaji because both address the same core training issue: augmentation strength improves generalization up to the point where it becomes label-destroying or otherwise harms learnability, and that threshold varies by sample. Shlens teaches learning an augmentation policy including operation selection and magnitude settings to improve object-detection performance, but it does not adapt augmentation strength on an instance-by-instance basis using training feedback. Balaji teaches that a single global strength parameter can be “too large for some images” and provides an epochwise, per-sample update rule for a scalar perturbation bound ei based on whether the model remains correct under the perturbed input, which is a direct hardness proxy. Incorporating Balaji’s instance-adaptive strength control into Shlens’ augmentation pipeline is a predictable variation that preserves Shlens’ operation-selection and bounding-box consistency while reducing harmful augmentations on fragile samples and increasing strength where tolerated. The expected result is improved detector performance and training stability with minimal implementation complexity because both systems already operate in iterative training loops with measured model outputs.
Regarding claim 2 and analogous claims 9 and 16:
Schlens and Balaji teach the method of claim 1.
Balaji teaches:
1. wherein the hardness is based on how far a respective prediction from the learning model is from a respective ground truth label.
(Balaji, pg. 3, Algorithm 2)
“If PGD succeeded in finding an image with a different class label, then ei is too big, so we replace ei ← ei − γ. If PGD failed, then we set ei ← ei + γ.”
One of ordinary skill in the art, at the time the invention was filed, would have been motivated to modify Schlens with Balaji. The motivation is the same as claim 1.
Regarding claim 4 and analogous claims 11 and 18:
Schlens and Balaji teach the method of claim 1.
Balaji teaches:
1. wherein measuring the hardness includes measuring the hardness of each data sample in the dataset.
(Balaji, pg. 3, Algorithm 2)
“If PGD succeeded in finding an image with a different class label, then ei is too big, so we replace ei ← ei − γ. If PGD failed, then we set ei ← ei + γ.”
PNG
media_image1.png
268
414
media_image1.png
Greyscale
One of ordinary skill in the art, at the time the invention was filed, would have been motivated to modify Schlens with Balaji. The motivation is the same as claim 1.
Regarding claim 5 and analogous claims 12 and 19:
Schlens and Balaji teach the method of claim 1.
Schlens teaches:
1. wherein adjusting the plurality of augmentations includes adjusting one or more augmentations for each data sample [based on corresponding hardness of each data sample in the dataset].
(Schlens, ¶0057)
“Thereafter, to generate the next iterative augmentation strategy 14, the controller model can perform evolutionary mutations on the augmentation strategy selected based on the comparison described above [i.e. wherein adjusting the plurality of augmentations includes adjusting one or more augmentations for each data sample].”
Ballaji teaches:
1. [wherein adjusting the plurality of augmentations includes adjusting one or more augmentations for each data sample] based on corresponding hardness of each data sample in the dataset.
(Balaji, pg. 3, Algorithm 2)
“If PGD succeeded in finding an image with a different class label, then ei is too big, so we replace ei ← ei − γ. If PGD failed, then we set ei ← ei + γ [i.e. based on corresponding hardness of each data sample in the dataset].”
One of ordinary skill in the art, at the time the invention was filed, would have been motivated to modify Schlens with Balaji. The motivation is the same as claim 1.
Regarding claim 7 and analogous claims 14 and 20:
Schlens and Balaji teach the method of claim 1.
Schlens teaches:
1. wherein adjusting the plurality of augmentations includes adjusting sampling ranges of one or more augmentations, based on the augmentation strength, by scaling upper and lower random sampling bounds of the sampling ranges.
(Schlens, ¶0089)
“FIG. 4C depicts an example augmented image 420 that results from applying a shift bounding shape operation to the image 300 of FIG. 3 in a Y direction [i.e. wherein adjusting the plurality of augmentations includes adjusting sampling ranges of one or more augmentations,]. In particular, as illustrated in FIG. 4C, the bounding shape 424 and the content 422 included in the bounding shape 424 has been shifted relative to the remainder of the image 420. A vacated space 426 can be filled with a fill color. The fill color can be a fixed, neutral color or can be a color that results from averaging the values of all pixels in the image [i.e. based on the augmentation strength, by scaling upper and lower random sampling bounds of the sampling ranges].”
One of ordinary skill in the art, at the time the invention was filed, would have been motivated to modify Schlens with Balaji. The motivation is the same as claim 1.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL JUSTIN BREENE whose telephone number is (571)272-6320. Examiner
interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-
based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO
Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on 303-297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786 9199 (IN USA OR CANADA) or 571-272-1000.
/P.J.B./ Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129