Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/22/2026 has been entered.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 18 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 18, “The system of claim 17” lacks antecedent basis in the claims.
Regarding claim 18, “The system of claim 17” appears to change the statutory category and it is unclear if claim 18 is to be a method or a system.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-2, 10, 12, and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Das (Das, Arun, and Paul Rad. "Opportunities and challenges in explainable artificial intelligence (xai): A survey." arXiv preprint arXiv:2006.11371 (2020) in view of Shiraki (WO2018235841A1).
Regarding claim 1, Das teaches an explainer system for explaining output of a prediction system comprising a system-monitor machine learning model trained to predict states of a monitored system (Algorithm 1, line 1, “f.predict(x)” where “f” can be a system-monitor machine learning model to predict states “y” of a monitored system), the explainer system comprising:
a computer-implemented perturbator configured to apply predetermined perturbations to original sample data collected from the monitored system to produce perturbed sample data (Algorithm 1, line 3, “Permute(x)”, §IV, “4) Local Interpretable Model-Agnostic Explanations (LIME)…Here for the input instance we permute data by finding a superpixel of information (‘fake’ data)” which constitutes applying a predetermined perturbation to original sample data from the monitored system to produce perturbed sample data)
the explainer system being configured to input the perturbed sample data to the prediction system (Algorithm 1, line 4, “f.predict(p)”);
a computer-implemented tester configured to receive model output from the prediction system (Algorithm 1, lines 5 and 7),
the model output comprising original model output produced by the system-monitor machine learning model based on the original sample data and deviated model output produced by the system-monitor machine learning model based on the perturbed sample data, the deviated model output comprising deviations from the original model output, the deviations resulting from the applied perturbations (Algorithm 1, line 5, “y” being output based on original sample data and “obs” being output based on the applie perturbations); and
an computer-implemented extractor configured to receive data defining the perturbations and the resulting deviations and to extract therefrom important features for explaining the model output (Algorithm 1, line 9), wherein
the computer-implemented extractor is configured to use an interpretable model to extract the important features for explaining the model output (Algorithm 1, line 9, “LinearModel.fit”) wherein
the computer-implemented tester is further configured to identify the deviations between the deviated model output and the original model output (Algorithm 1, line 5, “dist” identifies the deviations via distance function) and to map the identified deviations to the applied perturbations to provide mapped perturbed segment-deviation pairs as input data for the interpretable model (Algortihm 1, line 7, “simscore”).
Das does not teach the computer-implemented extractor configured to cause the important features to be presented via a user interface as an explanation of the output of the prediction system.
Shiraki teaches causing important features to be presented via a user interface as an explanation of the output of the prediction system (see “Device Configuration”, “Specifically, as shown in FIG. 4, the feature representation extraction unit 12 extracts the feature representation of the analysis target range from the learning data passing through the analysis target range using a chi-square test, and extracts the extracted feature representation Gives a score indicating the relevance to the analysis target range. In addition, as shown in FIG. 5, the feature representation extraction unit 12 extracts the feature representation of the comparison target range from the learning data passing through the comparison target range using the chi-square test, and extracts the extracted feature representation: Assign a score indicating the relevance to the comparison range… The display unit 13 displays the extracted feature representation for each of the analysis target range and the comparison target range. Specifically, the display unit 13 is extracted on the screen of a display device connected to the graph structure analysis device 10 or on the screen of a terminal device connected to the graph structure analysis device 10 via a network. Display feature representation. The display unit 13 also displays the difference when the feature expression extraction unit 12 obtains the difference of the feature expression.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify he computer-implemented extractor of Das to cause the important features to be presented via a user interface as an explanation of the output of the prediction system in order for users to ascertain the importance measures of various features and to evaluate the model (see “Industrial Applicability” in Shiraki).
PNG
media_image1.png
484
788
media_image1.png
Greyscale
Regarding claim 2, Das as modified teaches all of the limitations of claim 1, wherein
the computer-implemented perturbator is configured to determine the perturbations to be applied using machine learning and/or optimization (Algorithm 1, line 3, “Permute(x)” is a part of a machine learning workflow).
Regarding claim 10, Das as modified teaches all of the limitations of claim 1, wherein the original sample data comprises image data (Algorithm 1, line 3, “Randomly pick superpixels”, also see Figure 9).
Regarding claim 12, Das as modified teaches all of the limitations of claim 1, wherein
the original sample data comprises images (Algorithm 1, line 3, “Randomly pick superpixels”, also see Figure 9), wherein
the computer-implemented perturbator is configured to apply the perturbations using data augmentation techniques (Algorithm 1, line 3, “Randomly pick superpixels”).
Regarding claims 15-16, Das as modified according to claims 1-2 performs the method of claims 15-16, respectively, under normal operation. See MPEP 2112.02.
Claim(s) 3-5, 9, 17-18, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Das (Das, Arun, and Paul Rad. "Opportunities and challenges in explainable artificial intelligence (xai): A survey." arXiv preprint arXiv:2006.11371 (2020) in view of Shiraki (WO2018235841A1), further in view of Yoon (Yoon, Jihyeun, Kyungyul Kim, and Jongseong Jang. "Propagated perturbation of adversarial attack for well-known CNNs: Empirical study and its explanation." 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 2019.)
Regarding claim 3, Das as modified teaches all of the limitations of claim 2.
Das as modified does not teach wherein the computer-implemented perturbator comprises a computer-implemented search optimizer configured to use an iterative optimization algorithm whose objective function maximizes the deviation in output caused by candidate perturbations when perturbed sample data comprising the applied candidate perturbations are input to the system-monitor machine learning model, and wherein the computer-implemented perturbator is configured to apply the candidate perturbations determined by the computer-implemented search optimizer to be associated with the largest deviations.
Yoon teaches
the computer-implemented perturbator comprises a search optimizer configured to use an iterative optimization algorithm whose objective function maximizes the deviation in output caused by candidate perturbations when perturbed sample data comprising the applied candidate perturbations are input to the system-monitor machine learning model (see §2, “Fast gradient sign method” with objective function J(θ,x,y) where “The attacker continues to accumulate it until it successes to mis-classify for the classifier or gets to ϵ’s step limitation.” The deviation in output grows until a misclassification is produced, i.e. the deviation is maximized via the objective function to the point of misclassification), and wherein
the computer-implemented perturbator is configured to apply the candidate perturbations determined by the search optimizer to be associated with the largest deviations (§2, each of FGSM, i-FGSM, and mi-FGSM is used by the computer-implemented perturbator in Figure 2 as input to the predictor and per §2 each is maximized until misclassification is produced and therefore the candidate perturbations which produce the largest deviations are those provided to the model at runtime.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a computer-implemented search optimizer configured to use an iterative optimization algorithm whose objective function maximizes the deviation in output caused by candidate perturbations when perturbed sample data comprising the applied candidate perturbations are input to the system-monitor machine learning model, and wherein the computer-implemented perturbator is configured to apply the candidate perturbations determined by the computer-implemented search optimizer to be associated with the largest deviations in Das as taught by Yoon in order to increase accuracy (see §5 of Yoon).
Regarding claim 4, Das as modified teaches all of the limitations of claim 3, but does not teach wherein the computer-implemented search optimizer is configured, iteratively and until completion of the optimization: to generate one or more current-iteration candidate perturbations by modifying one or more previous-iteration candidate perturbations in accordance with the optimization algorithm, to provide perturbed sample data comprising the applied current-iteration candidate perturbations to the prediction system for input to the system-monitor machine learning model, and to receive, as feedback, deviated output produced by the system-monitor machine learning model based on the perturbed sample data, and to determine from the feedback a deviation caused by the current-iteration candidate perturbations.
Yoon teaches
the search optimizer is configured, iteratively and until completion of the optimization: to generate one or more current-iteration candidate perturbations by modifying one or more previous-iteration candidate perturbations in accordance with the optimization algorithm (§2, FGSM, for example, accumulates the gradient until misclassification occurs, i.e. it modifies previous perturbations by accumulation),
to provide perturbed sample data comprising the applied current-iteration candidate perturbations to the prediction system for input to the system-monitor machine learning model, (“The attacker continues to accumulate it until it successes to mis-classify for the classifier or gets to ϵ’s step limitation.” – current iterations are provided to determine whether misclassification has occurred) and
to receive, as feedback, deviated output produced by the system-monitor machine learning model based on the perturbed sample data, and to determine from the feedback a deviation caused by the current-iteration candidate perturbations (“The attacker continues to accumulate it until it successes to mis-classify for the classifier or gets to ϵ’s step limitation.” – the feedback being some mismatch between outputs and the determination being whether misclassification has occurred).
Das and Yoon are combinable under the same rational as claim 3.
Regarding claim 5, Das as modified teaches all of the limitations of claim 1, but does not teach one or more of: (i) a first perturbation selector machine learning model configured to receive as training data perturbed segment-deviation pairs and to learn to select perturbations that result in deviations in the model output of the system-monitor machine learning model that exceed deviations caused by other perturbations applied to the same original sample data; (ii) a second perturbation selector machine learning model configured to receive as training data the perturbed segment-deviation pairs and to learn to select perturbations that do not result in deviations in the model output that exceed deviations caused by other perturbations applied to the same original sample data.
Yoon teaches
a first perturbation selector machine learning model configured to receive as training data perturbed segment-deviation pairs and to learn to select perturbations that result in deviations in the model output of the system-monitor machine learning model that exceed deviations caused by other perturbations applied to the same original sample data (§2, i-FGSM, for example, accumulates the gradient until misclassification occurs, i.e. it selects perturbations which result exceeding deviations caused by other perturbations applied to the same original sample data and can be considered a machine learning model. One of the other models, e.g. FGSM or mi-FGSM, can be the computer-implemented perturbator of claim 1 and i-FGSM the perturbation selector machine learning model. The perturbations selected in claim 5 are not necessarily the predetermined perturbations of claim 1.)
Das and Yoon are combinable under the same rational as claim 3.
Regarding claim 9, Das as modified teaches all of the limitations of claim 1, but does not teach wherein the original sample data is un- preprocessed original sample data collected from the monitored system, and wherein the computer-implemented perturbator is configured to apply the perturbations to the un- preprocessed original sample data to produce un-preprocessed perturbed sample data, before the un-preprocessed perturbed sample data is formatted by a pre-processor to produce preprocessed perturbed sample data suitable for input to the system-monitor machine learning model.
Yoon teaches wherein
the original sample data is un-preprocessed original sample data collected from the monitored system, and wherein the computer-implemented perturbator is configured to apply the perturbations to the un-preprocessed original sample data to produce un-preprocessed perturbed sample data, before the un-preprocessed perturbed sample data is formatted by a pre-processor to produce preprocessed perturbed sample data suitable for input to the system-monitor machine learning model (see Figure 1. The original and perturbed images can be considered un-preprocessed and the Denoiser can be considered a preprocessor which prepares perturbed samples for input into the CNN).
Das and Yoon are combinable under the same rational as claim 3.
Regarding claims 17-18, Das as modified according to claims 3-4 performs the method of claims 17-18 under normal operation.
Regarding claim 19, Das as modified according to claim 5 performs the method of claim 19 under normal operation.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Das (Das, Arun, and Paul Rad. "Opportunities and challenges in explainable artificial intelligence (xai): A survey." arXiv preprint arXiv:2006.11371 (2020) in view of Shiraki (WO2018235841A1), further in view of Singh (Singh, Naman D., and Abhinav Dhall. "Clustering and learning from imbalanced data." arXiv preprint arXiv:1811.00972 (2018)).
Regarding claim 11, Das as modified teaches all of the limitations of claim 1, but does not teach wherein the computer-implemented perturbator is configured to apply the perturbations by oversampling the original sample data, the oversampling comprising clustering samples in the original sample data and generating new samples from within the clusters.
Singh discloses data perturbation via oversampling the original sample data, the oversampling comprising clustering samples in the original sample data and generating new samples from within the clusters (see §3, steps 1-6. Oversampling is performed from samples generated, i.e. perturbed samples, within the clusters and ensures more important samples have higher representation, “The way our algorithm decides on the number of new samples for each original instance makes sure that the more important samples have higher representation than far lying samples in the new balanced minority class space.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to configured to the computer-implemented perturbator to apply the perturbations by oversampling the original sample data, the oversampling comprising clustering samples in the original sample data and generating new samples from within the clusters in Das in order to provide perturbed samples which ensure that important samples are properly represented.
Allowable Subject Matter
Claims 6 and 8 are allowed.
Claims 7 and 20 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 7, the prior art does not support a prima facie case of obviousness or anticipation for the system of claim 5 comprising both models recited in claim 5 coupled with a perturbation finder which selects a perturbation from the outputs of each model.
Regarding claim 20, the reasons for allowance are the same as claim 6, mutatis mutandis. See Non-Final Rejection dated 01/14/2026.
Response to Arguments
Applicant’s remarks filed 06/22/2026 have been fully considered.
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SCHYLER S SANKS whose telephone number is (571)272-6125. The examiner can normally be reached 06:30 - 15:30 Central Time, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SCHYLER S SANKS/ Primary Examiner, Art Unit 2129
