Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-5, 9-10, and 12-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon (Yoon, Jihyeun, Kyungyul Kim, and Jongseong Jang. "Propagated perturbation of adversarial attack for well-known CNNs: Empirical study and its explanation." 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 2019.) in view of Shiraki (WO2018235841A1).
Regarding claim 1, Yoon teaches an explainer system for explaining output of a prediction system comprising a system-monitor machine learning model trained to predict states of a monitored system (see Figure 1 and Figure 2. Figure 1 shows an explainer system which inputs data into a prediction system (the CNN), and the CNN can be considered a system-monitor machine learning model which predicts states of a monitored system where the monitored system is the dataset it operates on), the explainer system comprising:
A computer-implemented perturbator configured to apply predetermined perturbations to original sample data collected from the monitored system to produce perturbed sample data (Figure 1, “Then, we also feed five types of adversaries and denoised adversaries to classifier for comparing the difference.”),
the explainer system being configured to input the perturbed sample data to the prediction system (Figure 1: “Configuration 1” and “Configuration 2” are input into the CNNs);
a computer-implemented tester configured to receive model output from the prediction system (see Figure 1,
v
o
r
i
l
a
n
d
v
a
d
v
l
are received for comparison),
the model output comprising original model output produced by the system-monitor machine learning model based on the original sample data and deviated model output produced by the system-monitor machine learning model based on the perturbed sample data, the deviated model output comprising deviations from the original model output, the deviations resulting from the applied perturbations (see Figure 1,
v
o
r
i
l
a
n
d
v
a
d
v
l
are received for comparison, “ori” indicates original output and “adv” indicates output from perturbed samples); and
an computer-implemented extractor configured to receive data defining the perturbations and the resulting deviations and to extract therefrom important features for explaining the model output (see Figure 2, where data defining the perturbations and the resulting deviations is utilized to provide an averaged normalized Euclidean distance between feature maps. Figure 3 provides the same for cosine distance).
Yoon does not teach a computer-implemented extractor configured to cause the important features to be presented via a user interface as an explanation of the output of the prediction system.
Shiraki teaches causing important features to be presented via a user interface as an explanation of the output of the prediction system (see “Device Configuration”, “Specifically, as shown in FIG. 4, the feature representation extraction unit 12 extracts the feature representation of the analysis target range from the learning data passing through the analysis target range using a chi-square test, and extracts the extracted feature representation Gives a score indicating the relevance to the analysis target range. In addition, as shown in FIG. 5, the feature representation extraction unit 12 extracts the feature representation of the comparison target range from the learning data passing through the comparison target range using the chi-square test, and extracts the extracted feature representation: Assign a score indicating the relevance to the comparison range… The display unit 13 displays the extracted feature representation for each of the analysis target range and the comparison target range. Specifically, the display unit 13 is extracted on the screen of a display device connected to the graph structure analysis device 10 or on the screen of a terminal device connected to the graph structure analysis device 10 via a network. Display feature representation. The display unit 13 also displays the difference when the feature expression extraction unit 12 obtains the difference of the feature expression.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify he computer-implemented extractor of Yoon to cause the important features to be presented via a user interface as an explanation of the output of the prediction system in order for users to ascertain the importance measures of various features and to evaluate the model (see “Industrial Applicability” in Shiraki).
Regarding claim 2, Yoon teaches all of the limitations of claim 1, wherein
the computer-implemented perturbator is configured to determine the perturbations to be applied using machine learning and/or optimization (§2, “Fast gradient sign method”, “Iterative fast gradient sign method”, and “Momentum iterative fast gradient sign method”).
Regarding claim 3, Yoon teaches all of the limitations of claim 1, wherein
the computer-implemented perturbator comprises a search optimizer configured to use an iterative optimization algorithm whose objective function maximizes the deviation in output caused by candidate perturbations when perturbed sample data comprising the applied candidate perturbations are input to the system-monitor machine learning model (see §2, “Fast gradient sign method” with objective function J(θ,x,y) where “The attacker continues to accumulate it until it successes to mis-classify for the classifier or gets to ϵ’s step limitation.” The deviation in output grows until a misclassification is produced, i.e. the deviation is maximized via the objective function to the point of misclassification), and wherein
the computer-implemented perturbator is configured to apply the candidate perturbations determined by the search optimizer to be associated with the largest deviations (§2, each of FGSM, i-FGSM, and mi-FGSM is used by the computer-implemented perturbator in Figure 2 as input to the predictor and per §2 each is maximized until misclassification is produced and therefore the candidate perturbations which produce the largest deviations are those provided to the model at runtime.)
Regarding claim 4, Yoon teaches all of the limitations of claim 3, wherein
the search optimizer is configured, iteratively and until completion of the optimization: to generate one or more current-iteration candidate perturbations by modifying one or more previous-iteration candidate perturbations in accordance with the optimization algorithm (§2, FGSM, for example, accumulates the gradient until misclassification occurs, i.e. it modifies previous perturbations by accumulation),
to provide perturbed sample data comprising the applied current-iteration candidate perturbations to the prediction system for input to the system-monitor machine learning model, (“The attacker continues to accumulate it until it successes to mis-classify for the classifier or gets to ϵ’s step limitation.” – current iterations are provided to determine whether misclassification has occurred) and
to receive, as feedback, deviated output produced by the system-monitor machine learning model based on the perturbed sample data, and to determine from the feedback a deviation caused by the current-iteration candidate perturbations (“The attacker continues to accumulate it until it successes to mis-classify for the classifier or gets to ϵ’s step limitation.” – the feedback being some mismatch between outputs and the determination being whether misclassification has occurred).
Regarding claim 5, Yoon teaches all of the limitations of claim 1, further comprising
a first perturbation selector machine learning model configured to receive as training data perturbed segment-deviation pairs and to learn to select perturbations that result in deviations in the model output of the system-monitor machine learning model that exceed deviations caused by other perturbations applied to the same original sample data (§2, i-FGSM, for example, accumulates the gradient until misclassification occurs, i.e. it selects perturbations which result exceeding deviations caused by other perturbations applied to the same original sample data and can be considered a machine learning model. One of the other models, e.g. FGSM or mi-FGSM, can be the computer-implemented perturbator of claim 1 and i-FGSM the perturbation selector machine learning model. The perturbations selected in claim 5 are not necessarily the predetermined perturbations of claim 1.)
Regarding claim 9, Yoon teaches all of the limitations of claim 1, wherein
the original sample data is un-preprocessed original sample data collected from the monitored system, and wherein the computer-implemented perturbator is configured to apply the perturbations to the un-preprocessed original sample data to produce un-preprocessed perturbed sample data, before the un-preprocessed perturbed sample data is formatted by a pre-processor to produce preprocessed perturbed sample data suitable for input to the system-monitor machine learning model (see Figure 1. The original and perturbed images can be considered un-preprocessed and the Denoiser can be considered a preprocessor which prepares perturbed samples for input into the CNN).
Regarding claim 10, Yoon teaches all of the limitations of claim 1, wherein the original sample data comprises image data (“datasets are generated by using TinyImageNet”).
Regarding claim 12, Yoon teaches all of the limitations of claim 1, wherein
the original sample data comprises images (“datasets are generated by using TinyImageNet”), wherein
the computer-implemented perturbator is configured to apply the perturbations using data augmentation techniques (§2, FGSM, for example).
Regarding claim 13, Yoon teaches all of the limitations of claim 1, wherein
the computer-implemented extractor is configured to use an interpretable model to extract the important features for explaining the model output (See Figure 2, interpretations are given, i.e. an “interpretable model” is used to extract important features for explaining model output).
Regarding claim 14, Yoon teaches all of the limitations of claim 13, wherein
the a computer-implemented tester is further configured to identify the deviations between the deviated model output and the original model output and to map the identified deviations to the applied perturbations to provide mapped perturbed segment-deviation pairs as input data for the interpretable model (see Figure 2, where Euclidean distance, i.e. identified deviation between deviated model output and original model output, is mapped with the adversarial perturbation applied to provide the mappings of Figure 2).
Regarding claims 15-19, Yoon according to claims 1-5 performs the method of claims 15-19, respectively, under normal operation. See MPEP 2112.02.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoon (Yoon, Jihyeun, Kyungyul Kim, and Jongseong Jang. "Propagated perturbation of adversarial attack for well-known CNNs: Empirical study and its explanation." 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 2019.) in view of Shiraki (WO2018235841A1), further in view of Singh (Singh, Naman D., and Abhinav Dhall. "Clustering and learning from imbalanced data." arXiv preprint arXiv:1811.00972 (2018)).
Regarding claim 11, Yoon as modified teaches all of the limitations of claim 1, but does not teach wherein the computer-implemented perturbator is configured to apply the perturbations by oversampling the original sample data, the oversampling comprising clustering samples in the original sample data and generating new samples from within the clusters.
Singh discloses data perturbation via oversampling the original sample data, the oversampling comprising clustering samples in the original sample data and generating new samples from within the clusters (see §3, steps 1-6. Oversampling is performed from samples generated, i.e. perturbed samples, within the clusters and ensures more important samples have higher representation, “The way our algorithm decides on the number of new samples for each original instance makes sure that the more important samples have higher representation than far lying samples in the new balanced minority class space.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to configured to the computer-implemented perturbator to apply the perturbations by oversampling the original sample data, the oversampling comprising clustering samples in the original sample data and generating new samples from within the clusters in Yoon in order to provide perturbed samples which ensure that important samples are properly represented.
Allowable Subject Matter
Claims 6 and 8 are allowed.
Claims 7 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 7, the prior art does not support a prima facie case of obviousness or anticipation for the system of claim 5 comprising both models recited in claim 5 coupled with a perturbation finder which selects a perturbation from the outputs of each model.
Regarding claim 20, the reasons for allowance are the same as claim 6, mutatis mutandis. See Non-Final Rejection dated 01/14/2026.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SCHYLER S SANKS whose telephone number is (571)272-6125. The examiner can normally be reached 06:30 - 15:30 Central Time, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SCHYLER S SANKS/ Primary Examiner, Art Unit 2129