Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments filed on 06/08/2026 have been entered. The status of the claims is as follows:
Claims 1-5, 7-13, 15-21, 23-27 remain pending in the application.
Claims 6, 14, 22 and 28 are cancelled.
Claims 1, 4-5, 9, 12-13, 17, 20-21 and 25 are amended.
Claims 26-28 are new.
Response to Arguments
In reference to the Claim Rejection under 35 U.S.C 103:
Argument 1:
Applicant argues in Remarks pg. 14-15 that Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, and further in view of Pan does not teach or suggest claim 1's element of "wherein the final feature set, the consistent feature set, and the differential feature set are each generated independently of classification model execution”
Response to Argument 1:
Applicant’s arguments are not persuasive. Under the BRI, the cited reference Hosseini teaches generating a final feature set based on both a differential feature set and a consistent feature set, independently of classification model execution. Specifically, Hosseini first evaluates the discriminative capability of features using the F statistic and ranks the features accordingly, which reasonably corresponds to generating a differential feature set. Hosseini then employs the KS statistical test to identify features having equal distributions, thereby identifying consistent (i.e., non-redundant) features. The reference further explains that “to form the final feature set, it is sufficient to select just one” of the feature having equal distributions, such that the final feature set is generated based on both the F-statistic ranking and the KS-based consistency analysis.(See Hosseini, Pg. 3, Col. 1, Section B, ¶[1-4]) Because these feature selection operations are performed as a preprocessing stage before the selected features are subsequently used for classification, the differential feature set, the consistent feature set, and the final feature set are each generated independently of classification model execution. The claim does not require the differential feature set and consistent feature set to be explicitly identified, stored, or output as separate intermediate data structures and such limitation are not read into the claim under BRI.
Applicant’s arguments filed 08/06/2026 have been fully considered but they are not persuasive.
Argument 2:
A second reason why claim 1 is not unpatentable over Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, and further in view of Pan is that Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, and further in view of Pan does not teach or suggest claim 1's element of "each dataset of the plurality of datasets comprises:... (iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry[.]" Hosseini, Hassani, Haws, and Rajagopal are each devoid of any teaching or suggestion of claim 1 's element of "each dataset of the plurality of datasets comprises:... (iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry
Response to Argument 2:
Applicant’s arguments have been considered but are not persuasive. The cited reference Pan et al. teaches training an adversarial classifier to distinguish between training data and testing data by predicting P ({train, test} | X), thereby assigning each individual entry a variable identifying whether the entry belongs to the training dataset or the testing dataset (see Pan, Pg. 3, Col. 1, ¶[1-3]). Under the BRI, the claimed “variable identifying a data source of the respective individual entry” reasonably encompasses a variable indicating the dataset from which each entry originates, such as the train/ test indicator disclosed by Pan. Thus, the reference teaches, or at least reasonably suggests, assigning to each individual training and testing entry a variable identifying its respective data source, as recited in the claim. Therefore, Applicant’s argument is not persuasive.
Applicant’s arguments filed 08/06/2026 have been fully considered but they are not persuasive.
Argument 3:
A third reason why claim is not unpatentable over Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, and further in view of Pan is that Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, and further in view of Pan does not teach or suggest claim 1's element of "wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries[.]" Hosseini, Hassani, Haws, and Pan are each devoid of any teaching or suggestion of claim 1's element of "wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries[.]
Response to Argument 3:
Applicant’s arguments have been considered but are not persuasive. The cited reference Pan et al. Teaches evaluating the statistical similarity of feature distributions between the training dataset and the testing dataset. Specifically, the reference discloses constructing a validation dataset by selecting training data such that the empirical distribution of the feature data is similar to the test data (i.e., PXval ≈ PXtest), thereby identifying statistical similar feature distribution between the datasets (see Pan, Pg. 3, Col. 1, Section 3.2). Pan further teaches that a close-to-random adversarial validation AUC score indicates a consistent feature distribution between the training and testing datasets. (See Pan, Pg. 4, Col. 2, Section 4.1) Additionally, the reference employs propensity score matching and evaluates the standardized mean difference (SMD) of covariates to determine whether the matched data ser statistically balanced (e.g., SMD < 0.1), thereby using a statistical measure to assess feature consistency between the training and testing data. Under the BRI, features satisfying these statistical similarity and balance criteria reasonably correspond to the claimed consistent feature set including features having statistical consistency between the set of training entries and the set of testing entries. Accordingly, the cited reference teaches, or at least reasonably suggests, the disputed limitation, and Applicant’s arguments are therefore not persuasive.
Applicant’s arguments filed 08/06/2026 have been fully considered but they are not persuasive.
Argument 4:
A fourth reason why claim is not unpatentable over Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, and further in view of Pan is that Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, and further in view of Pan does not teach or suggest claim 1's element of "partitioning... the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on the corresponding distribution test[.]"Hosseini, Rajagopal, Haws, and Pan are each devoid of any teaching or suggestion of claim l's element of "partitioning... the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on the corresponding distribution test[.]"
Applicant’s arguments filed 08/06/2026 have been fully considered but they are not persuasive.
Response to Argument 4:
Applicant’s arguments have been considered but are not persuasive. The cited reference Hassani & Silva teaches performing a two-sample Kolmogorov-Smirnov (KS) distribution test to determine whether two distributions are identical or different by testing the null hypothesis that the distributions are identical and the alternative hypothesis that the distributions differ. Under the BRI, applying the disclosed distribution test to predictive features inherently distinguishes features having statistically consistent distributions from features having statistically different distributions, thereby reasonably teaching partitioning the predictive features into a consistent feature set and a differential feature set based on the corresponding distribution test. Therefore, the cited reference teaches, or at least reasonably suggests, the disputed limitation.
Applicant’s arguments filed 08/06/2026 have been fully considered but they are not persuasive.
Argument 5:
Applicant asserts that Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, in view of Pan and further in view of Yu does not teach or suggest claim 3's element of "removing... the selected differential feature from the differential feature set in response to determining that the correlation value is outside the correlation range[.]" The Current Office Action appears to be asserting that "[f]eature Fj will be removed[.]" (Current Office Action, Page 99). However, this appears to be in the context of "removing its redundant peers... [i]f there exists such Fj to a feature Fi[.]" (Current Office Action, Page 66).
Response to Argument 5:
Applicant’s arguments have been considered but are not persuasive. The rejection does not equate “redundant” with “outside the correlation range”. Rather, the cited reference Yu & Liu determines whether a feature is redundant by computing a correlation value (SU) and comparing that correlation value against a predefined threshold. When the correlation value satisfies the redundancy criterion (i.e., exceeds or otherwise falls outside the acceptable correlation criterion defined by the threshold), one of the correlated features is removed from the selected feature subset. Under the BRI, the claimed “correlation range” encompasses such a predefined threshold or acceptable correlation criterion. Accordingly, the cited reference teaches removing a selected feature in response to determining that its correlation value is outside the claimed correlation range.
Applicant’s arguments filed 08/06/2026 have been fully considered but they are not persuasive.
Argument 6:
Applicant asserts that Hosseini, in view of Hassani, in view of Haws, in view of Rajagopal, in view of Pan and further in view of Yu does not teach or suggest claim 3's elements of "a correlation range of the set of training entities" and "a correlation value of the set of testing entities[.]" (Emphasis added). The Current Office Action asserts that both "the 'set of training entities' is being interpreted as 'class C"' and that "the 'set of testing entities' is being interpreted as 'class C[.]"' (Current Office Action, Pages 54-65). In contrast, claim 3 describes a "set of training entries" and a "set of testing entries[.]"
Response to Argument 6:
Applicant’s arguments have been fully considered. While Applicant asserts that the previous Office Action equated the claimed set of training entries with class C, the rejection has been clarified. Specifically, the cited reference is relied upon for teaching the determination of feature correlation values using symmetrical uncertainty (SU), including pairwise feature correlation (F-correlation), and comparing the determined correlation values against a predefined threshold to determine whether a feature should be retained or removed. Under the BRI, these teaching reasonably correspond to the claimed determination of correlation values and removal of features based on the correlation determination. Accordingly, as clarified below, the cited reference continues to teach, or at least reasonably suggest, the disputed limitation and Applicant’s arguments are therefore not persuasive.
Applicant’s arguments filed 08/06/2026 have been fully considered but they are not persuasive.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 4-5, 7-10, 12-13, 15-18, 20-21, 23-24 and 26-27 are rejected under 35 U.S.C. 103 as being unpatentable over Hosseini & Mahdavi (“F Plus KS: A new Feature Selection Strategy for Steganalysis”) (hereafter referred to as “Hosseini”) in view of Hassani & Silva (“A Kolmogorov-Smirnov Based Test for Comparing the Predictive Accuracy of Two Sets of Forecasts”) (hereafter referred to as “Hassani”) and further in view of Haws et al (US 9,483,739 B2) (hereafter referred to as “Haws”), Rajagopal et al. (“Towards Effective Network Intrusion Detection: From Concept to Creation on Azure Cloud”) (hereafter referred to as “Rajagopal”) and Pan et al. (“Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber”) (hereafter referred to as “Pan”)
Regarding claim 1, Hosseini explicitly discloses:
A computer-implemented method comprising: performing, by one or more
processors, on a plurality of datasets, a plurality of distribution tests, (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1), ¶[1-2]: “Kolmogorov-Smirnov (KS) is a statistical test evaluating the hypothesis that whether two independent sample sets represent two different population or not [16]. In KS-test, the maximum difference between cumulative probability distributions related to the two random variables is calculated. If there is a significant difference at any point along the two cumulative probability distributions, it can be concluded that the two sample sets belong to two different populations. The KS value between the two random variable M and N can be calculated from the following steps. Random variables outcomes x are discretized, separately, into k bins [xi, xi+1], I = 1 … k. Occurrence probabilities in each bin are estimated. Cumulative probability distributions are calculated. KS statistic can be calculated from (1).
PNG
media_image1.png
79
482
media_image1.png
Greyscale
In (1), nM and nN are the number of elements in M and N. Also Mk and Nk refer to cumulative probability distributions of M and N in kth bin, respectively. In each statistical significance level of α, if KS (M,N) < λα, the two random variables M and N are related to a unique distribution.”) [Examiner’s note: the “plurality of datasets” is being interpreted as the “cover” dataset and the “stegos” dataset. Because the features in these datasets are used to train a classifier to “predict suspect objects”, they are being interpreted as the “predictive features”. The “plurality of distribution tests” is being interpreted as the “cumulative probability distributions” and the “Kolmogorov-Smirnov statistical test”]
each dataset of the plurality of datasets comprises: (i) a set of training entries corresponding to a single predictive feature of the plurality of predictive features, (ii) a set of testing entries corresponding to the single predictive feature, and (Hosseini, Page 3, Col. 2, Section III, ¶[1-2]: “Quantity of 8,000 images were used as the cover images of training set and 2,000 remaining images were used as the cover images of testing set… In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain.”)
generating, [by one or more processors], a final feature set based on the differential feature set and the consistent feature set, wherein the final feature set, the consistent feature set, and the differential feature set are each generated independently of classification model execution. (Hosseini, Page 3, Col. 1, Section. B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them. In order to evaluate the similarity of the features, KS-test can be used and the evaluation can be made through a forward comparison”, Page 3, Col. 1, Section. B, ¶[3-4]: “To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section… The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”) [Examiner’s note: Hosseini discloses evaluating the similarity of the features by using KS test then sorting them to form the optimal final feature set, which aligns with the concept of partitioning the features into differential set (e.g., set of features which do not share similarity) and consistent set (e.g., set of features which share similarity)]
building, by one or more processors, a machine learning model using the final feature set; (Hosseini, Pg. 1, Col. 1, Section I, ¶[1]: “The statistical measures which model these relations are known as features”, Col. 2, ¶[1]: “Kodovský [6] introduced rich models which were combination of smaller sub-models made to capture inter-block and intra-block relationships, form both the image and calibrated one. The final feature set included 22,510 features.”)
Hosseini fails to disclose:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features;
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry;
the set of testing entries are devoid of a target;
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries
partitioning, [by one or more processors], the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on [[their]] the corresponding distribution test, wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries
However, Pan explicitly discloses:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features; (Pan, Abstract: “With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data.”, Pg. 2, Figure 2:
PNG
media_image2.png
295
572
media_image2.png
Greyscale
, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset. Classification performance better than random guess indicates the different feature distributions between the training and test datasets.”)
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry; (Pan, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset”, Pg. 3, Col. 1, ¶1]: “We start with a labeled training dataset {(ytrain, Xtrain)} ∈ R × Rd , and an unlabeled test dataset {Xtest } ∈ Rd with an unknown conditional probability Py |X . Then, we train an adversarial classifier that predicts P({train, test }|X) to separate train and test, and generate the propensity score ppropensity = P(test |X) on both Xtest and Xtrain.”) [Examiner’s note: Pan’s adversarial classifier is trained to predict whether a sample is from training or testing dataset, which is effectively a source label.]
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries (Pan, Pg. 3, Col. 1, ¶[2]: “The feature importance and propensity score from the adversarial classifier can be used to detect concept drift between the training and test data, and provide insights on the cause of the concept drift such as which features and subsamples in the training data are most different from ones in the test data.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Pan. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Pan teaches an adversarial validation approach to concept drift problems in user targeting automation systems. One of ordinary skill would have motivation to combine Hosseini and Pan in order to select, retain or remove features based not only on predictive usefulness but also on whether those features remain stable across datasets, thereby reducing selection bias, improving robustness to drift, and producing a model that generalizes better to unseen test data.
However, Rajagopal explicitly discloses:
and (iii) a data source variable indicating a type of each entry of the set of training entries and the set of testing entries; (Rajagopal, Pg. 19729, Table 2:
PNG
media_image3.png
338
553
media_image3.png
Greyscale
)
wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries (Rajagopal, Pg. 19732, Col. 1, ¶[5]: “While building machine learning models, it often becomes imperative to compare the performance of classifiers and the best way to achieve this is to perform statistical significance tests… In this context, the null hypothesis (H0) suggests that there is no performance difference among classifiers whereas an alternate hypothesis (H1) indicates that at least one classifier performs differently. Suppose, ‘d’ refers to the number of datasets and ‘k’ signifies the number of classifiers, Friedman test statistic can be calculated as shown in equation (7)
PNG
media_image4.png
63
460
media_image4.png
Greyscale
”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Rajagopal. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Rajagopal teaches a meta-classification approach using decision jungle to perform both binary and multiclass classification. One of ordinary skill would have motivation to combine Hosseini and Rajagopal because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Haws explicitly discloses:
the set of testing entries are devoid of a target; (Haws, Col. 1, Lines 53-58: “The feature selection module is configured to perform a method. The method includes receiving a set of training samples and a set of test samples. The set of training samples includes a first set of features and a class value. The set of test samples includes the set of features absent the class value.”) [Examiner’s note: the testing dataset is devoid of the target e.g., the set of test samples includes the set of features absent the class value]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Haws. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Haws teaches a transductive feature selection method with maximum-relevancy and minimum-redundancy criteria. One of ordinary skill would have motivation to combine Hosseini and Haws because by excluding the target outcome from the testing dataset during feature selection process, the features are chosen based purely on the underlying data patterns – not on knowledge of the final outcome. This leads to a more honest evaluation of the model’s true predictive power and helps create models that are reliable, and robust when applied to unseen data
However, Hassani explicitly discloses:
partitioning, [by one or more processors], the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on [[their]] the corresponding distribution test, (Hassani, Page 594, ¶[2]: “Next, we introduce the hypothesis which are relevant for the proposed KSPA test. Let us begin by presenting the hypothesis for the two-sided KS test. Let X and Y be two random variables with c.d.f.’s FX and FY , respectively. Then, a two sample, two-sided KS test will test the hypothesis that both c.d.f.’s have an identical distribution, and the resulting null and alternate hypothesis can be expressed as:
PNG
media_image5.png
76
410
media_image5.png
Greyscale
In simple terms, the null hypothesis in Equation (6) states that both X and Y share an identical distribution whilst the alternate hypothesis states that X and Y do not share the same distribution.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Hassani. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Hassani teaches performing a similarity statistical test between two different datasets using Kolmogorov-Smirnov technique. One of ordinary skill would have motivation to combine Hosseini and Hassani in order to measure how different the distributions of a feature are across groups. If a feature behaves very differently between two classes, that’s a good sign it’s useful for making predictions, this is especially helpful in real-world situations where data doesn’t follow neat patterns – since KS doesn’t assume any particular distributions, it works well no matter how messy the data gets.
Regarding claim 2, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of claim 1 (as shown in the rejection above).
Hosseini in view of Hassani, Haws, Rajagopal and Pan further discloses:
Identifying, by one or more processors, a lead feature from the consistent feature set; (Hosseini, Page 2, Col. 1, ¶[3]: “in the proposed method, F is used just for ranking features. Afterwards, a strategy based on Kolmogorov-Smirnov (KS) test [16] is used in order to remove redundant features.”, Page 3, Col. 1, Section. B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
)[Examiner’s note: Fig. 2 discloses the features are ranked based on F statistical measure in descending order, with the most important feature placed first, which aligns with the concept of identifying the lead feature.]
adjusting, by one or more processors, the differential feature set based on a correlation of each feature in the differential feature set to the lead feature; and (Hosseini, Page 3, Col. 1, Col. 1, Section B, ¶[3]: “Removing redundant features during forward comparison leads to sensitiveness of the algorithm to the ordering of the input features. To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section. In KS-CBF1 algorithm [19] which also uses KS for removing redundant features and is similar to the suggested algorithm, the features are reordered using a ranking system based on SUC2.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
) [Examiner’s note: The highlight indicates that the features that pass the KS test (i.e., have significantly different distribution) remain in the differential set, while features that fail the KS test (i.e., have nearly identical distribution to a higher-ranked feature) are removed. This means the algorithm in Fig. 2 is dynamically adjusting the differential set based on how similar each feature is to the lead feature]
generating, by one or more processors, the final feature set based on the adjusted differential feature set and the consistent feature set. (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”, Page 3, Col. 2, ¶[2]: “The result of comparing features is reduction of redundant features, which can lead to reduction of features that should be extracted from the image. Furthermore, the dimensionality of feature vectors will be decreased and as the result, classification complexity will be reduced.”) [Examiner’s note: the final feature set is generated after removing the redundant feature (i.e., adjusting the differential feature set), which results in a reduction of features.]
Regarding claim 4, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of claim 1 (as shown in the rejection above).
Hosseini in view of Hassani, Haws, Rajagopal and Pan further discloses:
wherein [[the]] partitioning the plurality of features further comprises: adding, by one or more processors, a first [[one]] predictive feature of the plurality of predictive features to the differential feature set in response to determining that the first predictive feature produces a first distribution corresponding to the set of training entries that is different from a second distribution corresponding to the set of testing entries. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 5, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of claim 4 (as shown in the rejection above).
Hosseini in view of Hassani, Haws, Rajagopal and Pan further discloses:
adding, by one or more processors, a second [[one]] predictive feature of the plurality of predictive features to the consistent feature set in response to determining that the second predictive feature produces a third distribution corresponding to the set of training entries that is equivalent to a fourth distribution corresponding to the set of testing entries. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 7, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of claim 1 (as shown in the rejection above).
Hosseini in view of Hassani, Haws, Rajagopal and Pan further discloses:
selecting, by one or more processors, a training dataset comprising a plurality of features and a target, wherein the training dataset comprises the set of training entries; (Hosseini, Page 3, Col. 2, Section III, ¶[1-2]: “Quantity of 8,000 images were used as the cover images of training set and 2,000 remaining images were used as the cover images of testing set… In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain.”)
for each of the plurality of features: selecting, by one or more processors, one of the
plurality of features; (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1)
performing, by one or more processors, a statistical test on the selected feature to
determine whether the selected feature is statistically important to the target; and (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1), ¶[1-2]: “Kolmogorov-Smirnov (KS) is a statistical test evaluating the hypothesis that whether two independent sample sets represent two different population or not [16]. In KS-test, the maximum difference between cumulative probability distributions related to the two random variables is calculated. If there is a significant difference at any point along the two cumulative probability distributions, it can be concluded that the two sample sets belong to two different populations. The KS value between the two random variable M and N can be calculated from the following steps. Random variables outcomes x are discretized, separately, into k bins [xi, xi+1], I = 1 … k. Occurrence probabilities in each bin are estimated. Cumulative probability distributions are calculated. KS statistic can be calculated from (1).
PNG
media_image1.png
79
482
media_image1.png
Greyscale
In (1), nM and nN are the number of elements in M and N. Also Mk and Nk refer to cumulative probability distributions of M and N in kth bin, respectively. In each statistical significance level of α, if KS (M,N) < λα, the two random variables M and N are related to a unique distribution.”) [Examiner’s note: the “plurality of datasets” is being interpreted as the “cover” dataset and the “stegos” dataset. Because the features in these datasets are used to train a classifier to “predict suspect objects”, they are being interpreted as the “predictive features”. The “plurality of distribution tests” is being interpreted as the “cumulative probability distributions” and the “Kolmogorov-Smirnov statistical test”]
adding, by one or more processors, the selected feature to the plurality of predictive features based on the statistical test. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 8, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of claim 7 (as shown in the rejections above).
Hosseini in view of Hassani, Haws, Rajagopal and Pan further discloses:
selecting, by one or more processors, a testing dataset comprising the plurality of features and which is devoid of the target, wherein the testing dataset comprises the set of testing entries; (Haws, Col. 1, Lines 53-58: “The feature selection module is configured to perform a method. The method includes receiving a set of training samples and a set of test samples. The set of training samples includes a first set of features and a class value. The set of test samples includes the set of features absent the class value.”) [Examiner’s note: the testing dataset is devoid of the target e.g., the set of test samples includes the set of features absent the class value]
selecting, by one or more processors, one of the plurality of predictive features; (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”)
combining, by one or more processors, one or more of the set of training entries corresponding to the selected predictive feature with one or more of the set of testing entries corresponding to the selected feature into a selected one of the plurality of datasets; and (Hosseini, Page 3, Col. 2, Section III, ¶[2]: “In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain. Then, a classification was performed where all the features were considered and the training and testing data were scaled according to the training data.”) [Examiner’s note: the highlights indicate that the features from both training set and testing set are extracted]
adding, by one or more processors, a data source variable to the selected dataset that indicates a dataset source of each of the entries in the selected dataset. (Hassani, Page 595, Section 2.2, ¶[1]: “Let us begin by defining forecast errors. Suppose we have a real valued, non zero time series YN = (y1, … yt, … yN) of sufficient length N. YN is divided into two parts, i.e., training set and test set such that Y1 = (y1, …, yt) represents the training set and Y2 = (yt+1,…, yN) represents the test set”) [Examiner’s note: “data source variable” is being interpreted as Y1 = (y1, …, yt) and Y2 = (yt+1, …, yN)]
Regarding claim 9, Hosseini explicitly discloses:
performing, on a plurality of datasets, a plurality of distribution tests, wherein (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1), ¶[1-2]: “Kolmogorov-Smirnov (KS) is a statistical test evaluating the hypothesis that whether two independent sample sets represent two different population or not [16]. In KS-test, the maximum difference between cumulative probability distributions related to the two random variables is calculated. If there is a significant difference at any point along the two cumulative probability distributions, it can be concluded that the two sample sets belong to two different populations. The KS value between the two random variable M and N can be calculated from the following steps. Random variables outcomes x are discretized, separately, into k bins [xi, xi+1], I = 1 … k. Occurrence probabilities in each bin are estimated. Cumulative probability distributions are calculated. KS statistic can be calculated from (1).
PNG
media_image1.png
79
482
media_image1.png
Greyscale
In (1), nM and nN are the number of elements in M and N. Also Mk and Nk refer to cumulative probability distributions of M and N in kth bin, respectively. In each statistical significance level of α, if KS (M,N) < λα, the two random variables M and N are related to a unique distribution.”) [Examiner’s note: the “plurality of datasets” is being interpreted as the “cover” dataset and the “stegos” dataset. Because the features in these datasets are used to train a classifier to “predict suspect objects”, they are being interpreted as the “predictive features”. The “plurality of distribution tests” is being interpreted as the “cumulative probability distributions” and the “Kolmogorov-Smirnov statistical test”]
each dataset of the plurality of datasets comprises: (i) a set of training entries corresponding to a single predictive feature of the plurality of predictive features, (ii) a set of testing entries corresponding to the single predictive feature, and (Hosseini, Page 3, Col. 2, Section III, ¶[1-2]: “Quantity of 8,000 images were used as the cover images of training set and 2,000 remaining images were used as the cover images of testing set… In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain.”)
generating a final feature set based on the differential feature set and the consistent feature set, wherein the final feature set, the consistent feature set, and the differential future set are each generated independently of classification model execution (Hosseini, Page 3, Col. 1, Section. B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them. In order to evaluate the similarity of the features, KS-test can be used and the evaluation can be made through a forward comparison”, Page 3, Col. 1, Section. B, ¶[3-4]: “To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section… The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”) [Examiner’s note: Hosseini discloses evaluating the similarity of the features by using KS test then sorting them to form the optimal final feature set, which aligns with the concept of partitioning the features into differential set (e.g., set of features which do not share similarity) and consistent set (e.g., set of features which share similarity)]
building, by one or more processors, a machine learning model using the final feature set; (Hosseini, Pg. 1, Col. 1, Section I, ¶[1]: “The statistical measures which model these relations are known as features”, Col. 2, ¶[1]: “Kodovský [6] introduced rich models which were combination of smaller sub-models made to capture inter-block and intra-block relationships, form both the image and calibrated one. The final feature set included 22,510 features.”)
Hosseini fails to disclose:
a processor set;
one or more computer-readable storage media; and
program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features;
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry;
the set of testing entries are devoid of a target;
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries
partitioning the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on the corresponding distribution test, wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries
However, Pan explicitly discloses:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features; (Pan, Abstract: “With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data.”, Pg. 2, Figure 2:
PNG
media_image2.png
295
572
media_image2.png
Greyscale
, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset. Classification performance better than random guess indicates the different feature distributions between the training and test datasets.”)
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry; (Pan, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset”, Pg. 3, Col. 1, ¶1]: “We start with a labeled training dataset {(ytrain, Xtrain)} ∈ R × Rd , and an unlabeled test dataset {Xtest } ∈ Rd with an unknown conditional probability Py |X . Then, we train an adversarial classifier that predicts P({train, test }|X) to separate train and test, and generate the propensity score ppropensity = P(test |X) on both Xtest and Xtrain.”) [Examiner’s note: Pan’s adversarial classifier is trained to predict whether a sample is from training or testing dataset, which is effectively a source label.]
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries (Pan, Pg. 3, Col. 1, ¶[2]: “The feature importance and propensity score from the adversarial classifier can be used to detect concept drift between the training and test data, and provide insights on the cause of the concept drift such as which features and subsamples in the training data are most different from ones in the test data.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Pan. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Pan teaches an adversarial validation approach to concept drift problems in user targeting automation systems. One of ordinary skill would have motivation to combine Hosseini and Pan in order to select, retain or remove features based not only on predictive usefulness but also on whether those features remain stable across datasets, thereby reducing selection bias, improving robustness to drift, and producing a model that generalizes better to unseen test data.
However, Rajagopal explicitly discloses:
and (iii) a data source variable indicating a type of each entry of the set of training entries and the set of testing entries; (Rajagopal, Pg. 19729, Table 2:
PNG
media_image3.png
338
553
media_image3.png
Greyscale
)
wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries (Rajagopal, Pg. 19732, Col. 1, ¶[5]: “While building machine learning models, it often becomes imperative to compare the performance of classifiers and the best way to achieve this is to perform statistical significance tests… In this context, the null hypothesis (H0) suggests that there is no performance difference among classifiers whereas an alternate hypothesis (H1) indicates that at least one classifier performs differently. Suppose, ‘d’ refers to the number of datasets and ‘k’ signifies the number of classifiers, Friedman test statistic can be calculated as shown in equation (7)
PNG
media_image4.png
63
460
media_image4.png
Greyscale
”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Rajagopal. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Rajagopal teaches a meta-classification approach using decision jungle to perform both binary and multiclass classification. One of ordinary skill would have motivation to combine Hosseini and Rajagopal because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Haws explicitly discloses:
a processor set; (Haws, Col. 1, Lines 49-51: “The information processing system includes a memory and a processor that is communicatively coupled to the memory)
one or more computer-readable storage media; and (Haws, Col. 1, Lines 49-51: “The information processing system includes a memory and a processor that is communicatively coupled to the memory)
program instructions stored [[in]] on the one or more computer-readable storage media to cause the processor set to perform operations comprising: (Haws, Col. 7, Lines 33-37: “In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Haws. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Haws teaches a transudative feature selection method with maximum-relevancy and minimum-redundancy criteria. One of ordinary skill would have motivation to combine Hosseini and Haws because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Hassani explicitly discloses:
partitioning the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on [[their]] the corresponding distribution test, (Hassani, Page 594, ¶[2]: “Next, we introduce the hypothesis which are relevant for the proposed KSPA test. Let us begin by presenting the hypothesis for the two-sided KS test. Let X and Y be two random variables with c.d.f.’s FX and FY , respectively. Then, a two sample, two-sided KS test will test the hypothesis that both c.d.f.’s have an identical distribution, and the resulting null and alternate hypothesis can be expressed as:
PNG
media_image5.png
76
410
media_image5.png
Greyscale
In simple terms, the null hypothesis in Equation (6) states that both X and Y share an identical distribution whilst the alternate hypothesis states that X and Y do not share the same distribution.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Hassani. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Hassani teaches performing a similarity statistical test between two different datasets using Kolmogorov-Smirnov technique. One of ordinary skill would have motivation to combine Hosseini and Hassani in order to measure how different the distributions of a feature are across groups. If a feature behaves very differently between two classes, that’s a good sign it’s useful for making predictions, this is especially helpful in real-world situations where data doesn’t follow neat patterns – since KS doesn’t assume any particular distributions, it works well no matter how messy the data gets.
Regarding claim 10, the combination of Hosseini, Hassani, Rajagopal, Haws and Pan discloses all the limitations of Claim 9 (as shown in the rejections above).
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
identifying a lead feature from the consistent feature set; (Hosseini, Page 2, Col. 1, ¶[3]: “in the proposed method, F is used just for ranking features. Afterwards, a strategy based on Kolmogorov-Smirnov (KS) test [16] is used in order to remove redundant features.”, Page 3, Col. 1, Section. B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
)[Examiner’s note: Fig. 2 discloses the features are ranked based on F statistical measure in descending order, with the most important feature placed first, which aligns with the concept of identifying the lead feature.]
adjusting the differential feature set based on a correlation of each feature in the differential feature set to the lead feature; and (Hosseini, Page 3, Col. 1, Col. 1, Section B, ¶[3]: “Removing redundant features during forward comparison leads to sensitiveness of the algorithm to the ordering of the input features. To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section. In KS-CBF1 algorithm [19] which also uses KS for removing redundant features and is similar to the suggested algorithm, the features are reordered using a ranking system based on SUC2.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
) [Examiner’s note: The highlight indicates that the features that pass the KS test (i.e., have significantly different distribution) remain in the differential set, while features that fail the KS test (i.e., have nearly identical distribution to a higher-ranked feature) are removed. This means the algorithm in Fig. 2 is dynamically adjusting the differential set based on how similar each feature is to the lead feature]
generating the final feature set based on the adjusted differential feature set and the consistent feature set. (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”, Page 3, Col. 2, ¶[2]: “The result of comparing features is reduction of redundant features, which can lead to reduction of features that should be extracted from the image. Furthermore, the dimensionality of feature vectors will be decreased and as the result, classification complexity will be reduced.”) [Examiner’s note: the final feature set is generated after removing the redundant feature (i.e., adjusting the differential feature set), which results in a reduction of features.]
Regarding claim 12, the combination of Hosseini, Hassani, Rajagopal, Haws and Pan discloses all the limitations of Claim 9 (as shown in the rejections above).
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
adding a first [[one]] predictive feature of the plurality of predictive features to the differential feature set in response to determining that the first predictive feature produces a first distribution corresponding to the set of training entries that is different from a second distribution corresponding to the set of testing entries. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 13, the combination of Hosseini, Hassani, Rajagopal, Haws and Pan discloses all the limitations of Claim 12 (as shown in the rejections above).
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
adding a second [[one]] predictive feature of the plurality of predictive features to the consistent feature set in response to determining that the second predictive feature produces a third distribution corresponding to the set of training entries that is equivalent to a fourth distribution corresponding to the set of testing entries. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 15, the combination of Hosseini, Hassani, Rajagopal, Haws and Pan discloses all the limitations of Claim 9 (as shown in the rejections above).
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
selecting a training dataset comprising a plurality of features and a target, wherein the training dataset comprises the set of training entries; (Hosseini, Page 3, Col. 2, Section III, ¶[1-2]: “Quantity of 8,000 images were used as the cover images of training set and 2,000 remaining images were used as the cover images of testing set… In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain.”)
for each of the plurality of features: selecting one of the plurality of features; (Hosseini,
Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1)
performing a statistical test on the selected feature to determine whether the selected feature is statistically important to the target; and (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1), ¶[1-2]: “Kolmogorov-Smirnov (KS) is a statistical test evaluating the hypothesis that whether two independent sample sets represent two different population or not [16]. In KS-test, the maximum difference between cumulative probability distributions related to the two random variables is calculated. If there is a significant difference at any point along the two cumulative probability distributions, it can be concluded that the two sample sets belong to two different populations. The KS value between the two random variable M and N can be calculated from the following steps. Random variables outcomes x are discretized, separately, into k bins [xi, xi+1], I = 1 … k. Occurrence probabilities in each bin are estimated. Cumulative probability distributions are calculated. KS statistic can be calculated from (1).
PNG
media_image1.png
79
482
media_image1.png
Greyscale
In (1), nM and nN are the number of elements in M and N. Also Mk and Nk refer to cumulative probability distributions of M and N in kth bin, respectively. In each statistical significance level of α, if KS (M,N) < λα, the two random variables M and N are related to a unique distribution.”) [Examiner’s note: the “plurality of datasets” is being interpreted as the “cover” dataset and the “stegos” dataset. Because the features in these datasets are used to train a classifier to “predict suspect objects”, they are being interpreted as the “predictive features”. The “plurality of distribution tests” is being interpreted as the “cumulative probability distributions” and the “Kolmogorov-Smirnov statistical test”]
adding the selected feature to the plurality of predictive features based on the statistical test. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 16, the combination of Hosseini, Hassani, Rajagopal, Haws and Pan discloses all the limitations of Claim 15 (as shown in the rejections above).
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
wherein the operations further comprises: selecting a testing dataset comprising the plurality of features and is devoid of the target, wherein the testing dataset comprises the set of testing entries; (Haws, Col. 1, Lines 53-58: “The feature selection module is configured to perform a method. The method includes receiving a set of training samples and a set of test samples. The set of training samples includes a first set of features and a class value. The set of test samples includes the set of features absent the class value.”) [Examiner’s note: the testing dataset is devoid of the target e.g., the set of test samples includes the set of features absent the class value]
selecting one of the plurality of predictive features; (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”)
combining at least one training entry of the set of training entries corresponding to the selected predictive feature with at least one testing entry of the set of testing entries corresponding to the selected feature into a selected dataset [[one]] of the plurality of datasets; and (Hosseini, Page 3, Col. 2, Section III, ¶[2]: “In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain. Then, a classification was performed where all the features were considered and the training and testing data were scaled according to the training data.”) [Examiner’s note: the highlights indicate that the features from both training set and testing set are extracted]
adding a data source variable to the selected dataset that indicates a dataset source of each of the entries in the selected dataset. (Hassani, Page 595, Section 2.2, ¶[1]: “Let us begin by defining forecast errors. Suppose we have a real valued, non zero time series YN = (y1, … yt, … yN) of sufficient length N. YN is divided into two parts, i.e., training set and test set such that Y1 = (y1, …, yt) represents the training set and Y2 = (yt+1,…, yN) represents the test set”) [Examiner’s note: “data source variable” is being interpreted as Y1 = (y1, …, yt) and Y2 = (yt+1, …, yN)]
Regarding claim 17, Hosseini explicitly discloses:
performing, on a plurality of datasets, a plurality of distribution tests, wherein (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1), ¶[1-2]: “Kolmogorov-Smirnov (KS) is a statistical test evaluating the hypothesis that whether two independent sample sets represent two different population or not [16]. In KS-test, the maximum difference between cumulative probability distributions related to the two random variables is calculated. If there is a significant difference at any point along the two cumulative probability distributions, it can be concluded that the two sample sets belong to two different populations. The KS value between the two random variable M and N can be calculated from the following steps. Random variables outcomes x are discretized, separately, into k bins [xi, xi+1], I = 1 … k. Occurrence probabilities in each bin are estimated. Cumulative probability distributions are calculated. KS statistic can be calculated from (1).
PNG
media_image1.png
79
482
media_image1.png
Greyscale
In (1), nM and nN are the number of elements in M and N. Also Mk and Nk refer to cumulative probability distributions of M and N in kth bin, respectively. In each statistical significance level of α, if KS (M,N) < λα, the two random variables M and N are related to a unique distribution.”) [Examiner’s note: the “plurality of datasets” is being interpreted as the “cover” dataset and the “stegos” dataset. Because the features in these datasets are used to train a classifier to “predict suspect objects”, they are being interpreted as the “predictive features”. The “plurality of distribution tests” is being interpreted as the “cumulative probability distributions” and the “Kolmogorov-Smirnov statistical test”]
each dataset of the plurality of datasets comprises: (i) a set of training entries corresponding to a single predictive feature of the plurality of predictive features, (ii) a set of testing entries corresponding to the single predictive feature, and (Hosseini, Page 3, Col. 2, Section III, ¶[1-2]: “Quantity of 8,000 images were used as the cover images of training set and 2,000 remaining images were used as the cover images of testing set… In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain.”)
generating a final feature set based on the differential feature set and the consistent feature set, wherein the final feature set, the consistent feature set, and the differential feature set are each generated independently of classification model execution. (Hosseini, Page 3, Col. 1, Section. B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them. In order to evaluate the similarity of the features, KS-test can be used and the evaluation can be made through a forward comparison”, Page 3, Col. 1, Section. B, ¶[3-4]: “To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section… The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”) [Examiner’s note: Hosseini discloses evaluating the similarity of the features by using KS test then sorting them to form the optimal final feature set, which aligns with the concept of partitioning the features into differential set (e.g., set of features which do not share similarity) and consistent set (e.g., set of features which share similarity)]
building, by one or more processors, a machine learning model using the final feature set; (Hosseini, Pg. 1, Col. 1, Section I, ¶[1]: “The statistical measures which model these relations are known as features”, Col. 2, ¶[1]: “Kodovský [6] introduced rich models which were combination of smaller sub-models made to capture inter-block and intra-block relationships, form both the image and calibrated one. The final feature set included 22,510 features.”)
Hosseini fails to disclose:
a processor set;
one or more computer-readable storage media; and
program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features;
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry;
the set of testing entries are devoid of a target;
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries
partitioning the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on [[their]] the corresponding distribution test, wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries
However, Pan explicitly discloses:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features; (Pan, Abstract: “With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data.”, Pg. 2, Figure 2:
PNG
media_image2.png
295
572
media_image2.png
Greyscale
, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset. Classification performance better than random guess indicates the different feature distributions between the training and test datasets.”)
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry; (Pan, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset”, Pg. 3, Col. 1, ¶1]: “We start with a labeled training dataset {(ytrain, Xtrain)} ∈ R × Rd , and an unlabeled test dataset {Xtest } ∈ Rd with an unknown conditional probability Py |X . Then, we train an adversarial classifier that predicts P({train, test }|X) to separate train and test, and generate the propensity score ppropensity = P(test |X) on both Xtest and Xtrain.”) [Examiner’s note: Pan’s adversarial classifier is trained to predict whether a sample is from training or testing dataset, which is effectively a source label.]
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries (Pan, Pg. 3, Col. 1, ¶[2]: “The feature importance and propensity score from the adversarial classifier can be used to detect concept drift between the training and test data, and provide insights on the cause of the concept drift such as which features and subsamples in the training data are most different from ones in the test data.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Pan. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Pan teaches an adversarial validation approach to concept drift problems in user targeting automation systems. One of ordinary skill would have motivation to combine Hosseini and Pan in order to select, retain or remove features based not only on predictive usefulness but also on whether those features remain stable across datasets, thereby reducing selection bias, improving robustness to drift, and producing a model that generalizes better to unseen test data.
However, Rajagopal explicitly discloses:
and (iii) a data source variable indicating a type of each entry of the set of training entries and the set of testing entries; (Rajagopal, Pg. 19729, Table 2:
PNG
media_image3.png
338
553
media_image3.png
Greyscale
)
wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries (Rajagopal, Pg. 19732, Col. 1, ¶[5]: “While building machine learning models, it often becomes imperative to compare the performance of classifiers and the best way to achieve this is to perform statistical significance tests… In this context, the null hypothesis (H0) suggests that there is no performance difference among classifiers whereas an alternate hypothesis (H1) indicates that at least one classifier performs differently. Suppose, ‘d’ refers to the number of datasets and ‘k’ signifies the number of classifiers, Friedman test statistic can be calculated as shown in equation (7)
PNG
media_image4.png
63
460
media_image4.png
Greyscale
”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Rajagopal. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Rajagopal teaches a meta-classification approach using decision jungle to perform both binary and multiclass classification. One of ordinary skill would have motivation to combine Hosseini and Rajagopal because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Haws explicitly discloses:
a processor set; (Haws, Col. 1, Lines 49-51: “The information processing system includes a memory and a processor that is communicatively coupled to the memory)
one or more computer-readable storage media; and (Haws, Col. 1, Lines 49-51: “The information processing system includes a memory and a processor that is communicatively coupled to the memory)
program instructions stored [[in]] on the one or more computer-readable storage media to cause the processor set to perform operations comprising: (Haws, Col. 7, Lines 33-37: “In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Haws. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Haws teaches a transudative feature selection method with maximum-relevancy and minimum-redundancy criteria. One of ordinary skill would have motivation to combine Hosseini and Haws because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Hassani explicitly discloses:
partitioning the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on [[their]] the corresponding distribution test, (Hassani, Page 594, ¶[2]: “Next, we introduce the hypothesis which are relevant for the proposed KSPA test. Let us begin by presenting the hypothesis for the two-sided KS test. Let X and Y be two random variables with c.d.f.’s FX and FY , respectively. Then, a two sample, two-sided KS test will test the hypothesis that both c.d.f.’s have an identical distribution, and the resulting null and alternate hypothesis can be expressed as:
PNG
media_image5.png
76
410
media_image5.png
Greyscale
In simple terms, the null hypothesis in Equation (6) states that both X and Y share an identical distribution whilst the alternate hypothesis states that X and Y do not share the same distribution.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Hassani. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Hassani teaches performing a similarity statistical test between two different datasets using Kolmogorov-Smirnov technique. One of ordinary skill would have motivation to combine Hosseini and Hassani in order to measure how different the distributions of a feature are across groups. If a feature behaves very differently between two classes, that’s a good sign it’s useful for making predictions, this is especially helpful in real-world situations where data doesn’t follow neat patterns – since KS doesn’t assume any particular distributions, it works well no matter how messy the data gets.
Regarding claim 18, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of Claim 17 (as shown in the rejections above)
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
wherein the operations further comprise: identifying a lead feature from the consistent feature set; (Hosseini, Page 2, Col. 1, ¶[3]: “in the proposed method, F is used just for ranking features. Afterwards, a strategy based on Kolmogorov-Smirnov (KS) test [16] is used in order to remove redundant features.”, Page 3, Col. 1, Section. B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
)[Examiner’s note: Fig. 2 discloses the features are ranked based on F statistical measure in descending order, with the most important feature placed first, which aligns with the concept of identifying the lead feature.]
adjusting the differential feature set based on a correlation of each feature in the differential feature set to the lead feature; and (Hosseini, Page 3, Col. 1, Col. 1, Section B, ¶[3]: “Removing redundant features during forward comparison leads to sensitiveness of the algorithm to the ordering of the input features. To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section. In KS-CBF1 algorithm [19] which also uses KS for removing redundant features and is similar to the suggested algorithm, the features are reordered using a ranking system based on SUC2.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
) [Examiner’s note: The highlight indicates that the features that pass the KS test (i.e., have significantly different distribution) remain in the differential set, while features that fail the KS test (i.e., have nearly identical distribution to a higher-ranked feature) are removed. This means the algorithm in Fig. 2 is dynamically adjusting the differential set based on how similar each feature is to the lead feature]
generating the final feature set based on the adjusted differential feature set and the consistent feature set. (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”, Page 3, Col. 2, ¶[2]: “The result of comparing features is reduction of redundant features, which can lead to reduction of features that should be extracted from the image. Furthermore, the dimensionality of feature vectors will be decreased and as the result, classification complexity will be reduced.”) [Examiner’s note: the final feature set is generated after removing the redundant feature (i.e., adjusting the differential feature set), which results in a reduction of features.]
Regarding claim 20, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of Claim 17 (as shown in the rejections above)
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
adding a first [[one]] predictive feature of the plurality of predictive features to the differential feature set in response to determining that the first predictive feature produces a first distribution corresponding to the set of training entries that is different from a second distribution corresponding to the set of testing entries. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 21, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of Claim 20 (as shown in the rejections above)
Hosseini in view of Hassani, Rajagoapl, Haws and Pan further discloses:
wherein the operations further comprise: adding a second [[one]] predictive feature of the plurality of predictive features to the consistent feature set in response to determining that the second predictive feature produces a third distribution corresponding to the set of training entries that is equivalent to a fourth distribution corresponding to the set of testing entries. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 23, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of Claim 17 (as shown in the rejections above)
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
selecting a training dataset comprising a plurality of features and a target, wherein the training dataset comprises the set of training entries; (Hosseini, Page 3, Col. 2, Section III, ¶[1-2]: “Quantity of 8,000 images were used as the cover images of training set and 2,000 remaining images were used as the cover images of testing set… In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain.”)
for each of the plurality of features: selecting one of the plurality of features; (Hosseini,
Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1)
performing a statistical test on the selected feature to determine whether the selected feature is statistically important to the target; and (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1), ¶[1-2]: “Kolmogorov-Smirnov (KS) is a statistical test evaluating the hypothesis that whether two independent sample sets represent two different population or not [16]. In KS-test, the maximum difference between cumulative probability distributions related to the two random variables is calculated. If there is a significant difference at any point along the two cumulative probability distributions, it can be concluded that the two sample sets belong to two different populations. The KS value between the two random variable M and N can be calculated from the following steps. Random variables outcomes x are discretized, separately, into k bins [xi, xi+1], I = 1 … k. Occurrence probabilities in each bin are estimated. Cumulative probability distributions are calculated. KS statistic can be calculated from (1).
PNG
media_image1.png
79
482
media_image1.png
Greyscale
In (1), nM and nN are the number of elements in M and N. Also Mk and Nk refer to cumulative probability distributions of M and N in kth bin, respectively. In each statistical significance level of α, if KS (M,N) < λα, the two random variables M and N are related to a unique distribution.”) [Examiner’s note: the “plurality of datasets” is being interpreted as the “cover” dataset and the “stegos” dataset. Because the features in these datasets are used to train a classifier to “predict suspect objects”, they are being interpreted as the “predictive features”. The “plurality of distribution tests” is being interpreted as the “cumulative probability distributions” and the “Kolmogorov-Smirnov statistical test”]
adding the selected feature to the plurality of predictive features based on the statistical test. (Hassani, Page 595, Section 2.2, ¶[3]: “The forecast errors in (13) or (14) are inputs into the KSPA test for determining the existence of a statistically significant difference in the distribution of forecasts from models m1 and m2. As the requirement is to test the distribution between two samples of forecast errors, the two sample two-sided KSPA test statistic can be calculated as:
PNG
media_image7.png
52
345
media_image7.png
Greyscale
where
F
ε
i
+
h
m
1
and
F
ε
i
+
h
m
2
denote the empirical c.d.f.’s for the forecast errors from two different models.”, Page 596, ¶[2]: “Accordingly, in terms of forecast errors, the two-sided KSPA test hypothesis can be approximately represented as follows, where
ε
i
+
h
m
1
and
ε
i
+
h
m
2
are the absolute or squared forecast errors from two forecasting models m1 and m2 with unknown continuous empirical c.d.f’s, the two-sided KSPA test will test the hypothesis:
PNG
media_image8.png
102
276
media_image8.png
Greyscale
Then, if the observed significance value of the two-sample two-sided KSPA test statistic Di, i+h is less than α (which is usually considered at the 1%, 5% or 10% level), we reject the null hypothesis and accept the alternate which is that the forecast errors
ε
i
+
h
m
1
and
ε
i
+
h
m
2
do not share the same distribution.”) [Examiner’s note: The null hypothesis H0 states that both model set share identical distribution (i.e., consistent set), the alternate hypothesis H1 states that they do not share the same distribution (i.e., differential set). The “two forecasting models m1 and m2” is being interpreted as the “training dataset” and “testing dataset”]
Regarding claim 24, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of Claim 23 (as shown in the rejections above)
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
selecting a testing dataset comprising the plurality of features and is devoid of the target, wherein the testing dataset comprises the set of testing entries; (Haws, Col. 1, Lines 53-58: “The feature selection module is configured to perform a method. The method includes receiving a set of training samples and a set of test samples. The set of training samples includes a first set of features and a class value. The set of test samples includes the set of features absent the class value.”) [Examiner’s note: the testing dataset is devoid of the target e.g., the set of test samples includes the set of features absent the class value]
selecting one of the plurality of predictive features; (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”)
combining at least one training entry of the set of training entries corresponding to the selected predictive feature with at least one testing entry of the set of testing entries corresponding to the selected feature into a selected dataset [[one]] of the plurality of datasets; and (Hosseini, Page 3, Col. 2, Section III, ¶[2]: “In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain. Then, a classification was performed where all the features were considered and the training and testing data were scaled according to the training data.”) [Examiner’s note: the highlights indicate that the features from both training set and testing set are extracted]
adding a data source variable to the selected dataset that indicates a dataset source of each of the entries in the selected dataset. (Hassani, Page 595, Section 2.2, ¶[1]: “Let us begin by defining forecast errors. Suppose we have a real valued, non zero time series YN = (y1, … yt, … yN) of sufficient length N. YN is divided into two parts, i.e., training set and test set such that Y1 = (y1, …, yt) represents the training set and Y2 = (yt+1,…, yN) represents the test set”) [Examiner’s note: “data source variable” is being interpreted as Y1 = (y1, …, yt) and Y2 = (yt+1, …, yN)]
Regarding claim 26, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of Claim 1 (as shown in the rejections above)
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
(New) The computer-implemented method of claim 1, further comprising: generating, by one or more processors, the plurality of predictive features by performing filter-based feature selection on a training dataset. (Pan, Pg. 3, Section 3.1, ¶[1-2]: “However, if the adversarial classifier can distinguish between training and test data well (i.e. AUC score ≫ 50%), the top features from the adversarial classifier are potential candidates exhibiting concept drift between the train and test data. We can then exclude these features from model training. Such feature selection can be automated by determining the number of features to exclude based on the performance of adversarial classifier (e.g. AUC score) and raw feature importance values (e.g. mean decrease impurity (MDI) in Decision Trees) as follows:”)
Regarding claim 27, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of Claim 1 (as shown in the rejections above)
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
(New) The computer-implemented method of claim 1, wherein the variable indicates whether the respective individual entry is of a selection from the group consisting of: the set of testing entries and the set of training entries. (Pan, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset”, Pg. 3, Col. 1, ¶1]: “We start with a labeled training dataset {(ytrain, Xtrain)} ∈ R × Rd , and an unlabeled test dataset {Xtest } ∈ Rd with an unknown conditional probability Py |X . Then, we train an adversarial classifier that predicts P({train, test }|X) to separate train and test, and generate the propensity score ppropensity = P(test |X) on both Xtest and Xtrain.”) [Examiner’s note: Pan’s adversarial classifier is trained to predict whether a sample is from training or testing dataset, which is effectively a source label.]
Claim(s) 3 is rejected under 35 U.S.C. 103 as being unpatentable over Hosseini & Mahdavi (“F Plus KS: A new Feature Selection Strategy for Steganalysis”) (hereafter referred to as “Hosseini”) in view of Hassani & Silva (“A Kolmogorov-Smirnov Based Test for Comparing the Predictive Accuracy of Two Sets of Forecasts”) (hereafter referred to as “Hassani”), Haws et al (US 9,483,739 B2) (hereafter referred to as “Haws”), Rajagopal et al. (“Towards Effective Network Intrusion Detection: From Concept to Creation on Azure Cloud”) (hereafter referred to as “Rajagopal”), Pan et al. (“Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber”) (hereafter referred to as “Pan”) and further in view of Yu & Liu (“Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution”) (hereafter referred to as “Yu”)
Regarding claim 3, the combination of Hosseini, Haws, Rajagopal, Hassani and Pan discloses all the limitations of claim 2 (as shown in the rejection above).
Hosseini in view of Hassani, Haws, Rajagopal and Pan further discloses:
wherein the adjusting the differential feature set further comprises: selecting, by
one or more processors, one of the features in the differential feature set; (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”, Page 3, Col. 1, Section B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”)
Hosseini in view of Hassani, Haws and Rajagopal fails to discloses:
computing, by one or more processors, a correlation range of the set of training entities based on a correlation between a set of first data values of the set of training entities and the lead feature;
computing, by one or more processors, a correlation value of the set of testing entities based on a correlation between a set of second data values of the set of testing entities and the lead feature; and
removing, by one or more processors, the selected differential feature from the differential feature set in response to determining that the correlation value is outside the correlation range
However, Yu explicitly discloses:
computing, by one or more processors, a correlation range of the set of training entities based on a correlation between a set of first data values of the set of training entities and the lead feature; (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”, Pg. 5, Section 4.2: “input: S(F1, F2,… FN, C) // a training data set”)
computing, by one or more processors, a correlation value of the set of testing entities based on a correlation between a set of second data values of the set of testing entities and the lead feature; and (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: the “lead feature” is being interpreted as the predominant feature Fi, and the “set of testing entities” is being interpreted as “class C” because class C represents the target values that we are trying to predict]
removing, by one or more processors, the selected differential feature from the differential feature set in response to determining that the correlation value is outside the correlation range. (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that
∀
Fi
∈
S’, 1
≤
i
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, Definition 2: “A feature is predominant to the class, iff its correlation to the class is predominant or can become predominant after removing its redundant peers.”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: “the correlation range” is being interpreted as the “threshold SU value ᵹ”. Feature Fj will be removed when its threshold value SU is larger than (i.e., outside the correlation range) the threshold value SU of feature Fi]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini, Hassani, Haws, Rajagopal and Yu. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Hassani teaches performing a similarity statistical test between two different datasets using Kolmogorov-Smirnov technique. Rajagopal teaches a meta-classification approach using decision jungle to perform both binary and multiclass classification. Haws teaches a transudative feature selection method with maximum-relevancy and minimum-redundancy criteria. Yu teaches a correlation-based feature selection method. One of ordinary skill would have motivation to combine Hosseini, Hassani, Haws, Rajagopal and Yu in order to identify redundancy and keep the model focused on the most valuable information. If two features are highly correlated, they are basically telling the model the same thing. Including both doesn’t add value – it just clutters the model, increases complexity, and can even lead to overfitting. By computing the correlation range, you can spots these overlaps early and remove redundant features.
Claim(s) 11, 19 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Hosseini & Mahdavi (“F Plus KS: A new Feature Selection Strategy for Steganalysis”) (hereafter referred to as “Hosseini”) in view of Hassani & Silva (“A Kolmogorov-Smirnov Based Test for Comparing the Predictive Accuracy of Two Sets of Forecasts”) (hereafter referred to as “Hassani”) , Haws et al (US 9,483,739 B2) (hereafter referred to as “Haws”), Rajagopal et al. (“Towards Effective Network Intrusion Detection: From Concept to Creation on Azure Cloud”) (hereafter referred to as “Rajagopal”), Pan et al. (“Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber”) (hereafter referred to as “Pan”) and further in view of Yu & Liu (“Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution”) ) (hereafter referred to as “Yu”)
Regarding claim 11, the combination of Hosseini, Hassani, Haws and Pan discloses all the limitations of claim 10 (as shown in the rejections above).
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
selecting one of the features in the differential feature set; (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”, Page 3, Col. 1, Section B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”)
Hosseini in view of Hassani, Rajagopal, Haws and Pan fails to discloses:
computing a correlation range of the set of training entries based on a correlation between a set of first data values of the set of training entries and the lead feature;
computing a correlation value of the set of testing entries based on a correlation between a set of second data values of the set of testing entries and the lead feature; and
removing the selected differential feature from the differential feature set in response to determining that the correlation value is outside the correlation range
However, Yu explicitly discloses:
computing a correlation range of the set of training entries based on a correlation between a set of first data values of the set of training entries and the lead feature; (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: the “lead feature” is being interpreted as the predominant feature Fi, and the “set of training entities” is being interpreted as “class C” because class C represents the target values that we are trying to predict]
computing a correlation value of the set of testing entries based on a correlation between a set of second data values of the set of testing entries and the lead feature; and (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: the “lead feature” is being interpreted as the predominant feature Fi, and the “set of testing entities” is being interpreted as “class C” because class C represents the target values that we are trying to predict]
removing the selected differential feature from the differential feature set in response to determining that the correlation value is outside the correlation range. (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that
∀
Fi
∈
S’, 1
≤
i
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, Definition 2: “A feature is predominant to the class, iff its correlation to the class is predominant or can become predominant after removing its redundant peers.”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: “the correlation range” is being interpreted as the “threshold SU value ᵹ”. Feature Fj will be removed when its threshold value SU is larger than (i.e., outside the correlation range) the threshold value SU of feature Fi]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini, Hassani, Rajagopal, Haws, Pan and Yu. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Hassani teaches performing a similarity statistical test between two different datasets using Kolmogorov-Smirnov technique. Haws teaches a transudative feature selection method with maximum-relevancy and minimum-redundancy criteria. Rajagopal teaches a meta-classification approach using decision jungle to perform both binary and multiclass classification. Pan teaches an adversarial validation approach to concept drift problems in user targeting automation systems. Yu teaches a correlation-based feature selection method. One of ordinary skill would have motivation to combine Hosseini, Hassani, Rajagopal, Haws, Pan and Yu in order to identify redundancy and keep the model focused on the most valuable information. If two features are highly correlated, they are basically telling the model the same thing. Including both doesn’t add value – it just clutters the model, increases complexity, and can even lead to overfitting. By computing the correlation range, you can spots these overlaps early and remove redundant features.
Regarding claim 19, the combination of Hosseini, Hassani, Rajagopal, Haws and Pan discloses all the limitations of claim 18 (as shown in the rejections above).
Hosseini in view of Hassani, Rajagopal, Haws and Pan further discloses:
selecting one of the features in the differential feature set; (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”, Page 3, Col. 1, Section B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”)
Hosseini in view of Hassani, Rajagopal, Haws and Pan fails to discloses:
computing a correlation range of the set of training entries based on a correlation between a set of first data values of the set of training entries and the lead feature;
computing a correlation value of the set of testing entries based on a correlation between a set of second data values of the set of testing entries and the lead feature; and
removing the selected differential feature from the differential feature set in response to determining that the correlation value is outside the correlation range
However, Yu explicitly discloses:
computing a correlation range of the set of training entries based on a correlation between a set of first data values of the set of training entries and the lead feature; (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: the “lead feature” is being interpreted as the predominant feature Fi, and the “set of training entities” is being interpreted as “class C” because class C represents the target values that we are trying to predict]
computing a correlation value of the set of testing entries based on a correlation between a set of second data values of the set of testing entries and the lead feature; and (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: the “lead feature” is being interpreted as the predominant feature Fi, and the “set of testing entities” is being interpreted as “class C” because class C represents the target values that we are trying to predict]
removing the selected differential feature from the differential feature set in response to determining that the correlation value is outside the correlation range. (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that
∀
Fi
∈
S’, 1
≤
i
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, Definition 2: “A feature is predominant to the class, iff its correlation to the class is predominant or can become predominant after removing its redundant peers.”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: “the correlation range” is being interpreted as the “threshold SU value ᵹ”. Feature Fj will be removed when its threshold value SU is larger than (i.e., outside the correlation range) the threshold value SU of feature Fi]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini, Hassani, Rajagopal, Haws, Pan and Yu. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Hassani teaches performing a similarity statistical test between two different datasets using Kolmogorov-Smirnov technique. Haws teaches a transudative feature selection method with maximum-relevancy and minimum-redundancy criteria. Rajagopal teaches a meta-classification approach using decision jungle to perform both binary and multiclass classification. Pan teaches an adversarial validation approach to concept drift problems in user targeting automation systems. Yu teaches a correlation-based feature selection method. One of ordinary skill would have motivation to combine Hosseini, Hassani, Rajagopal, Haws, Pan and Yu in order to identify redundancy and keep the model focused on the most valuable information. If two features are highly correlated, they are basically telling the model the same thing. Including both doesn’t add value – it just clutters the model, increases complexity, and can even lead to overfitting. By computing the correlation range, you can spots these overlaps early and remove redundant features.
Regarding claim 25, Hosseini explicitly discloses:
performing, by one or more processors, on a plurality of datasets, a plurality of distribution tests, wherein (Hosseini, Page 1, Col. 1, Section I, ¶[2]: “In steganalysis, the digital object that does not contain embedded data is named cover and the one that contains embedded data is named stego. In order to make a decision about suspect objects, a classifier is trained with the features extracted from sufficient number of covers and stegos [1]; Then, the classifier predicts on the suspect objects and decides whether each of them contains hidden data or not.”, Page 2, Col. 1, Section II.A.1), ¶[1-2]: “Kolmogorov-Smirnov (KS) is a statistical test evaluating the hypothesis that whether two independent sample sets represent two different population or not [16]. In KS-test, the maximum difference between cumulative probability distributions related to the two random variables is calculated. If there is a significant difference at any point along the two cumulative probability distributions, it can be concluded that the two sample sets belong to two different populations. The KS value between the two random variable M and N can be calculated from the following steps. Random variables outcomes x are discretized, separately, into k bins [xi, xi+1], I = 1 … k. Occurrence probabilities in each bin are estimated. Cumulative probability distributions are calculated. KS statistic can be calculated from (1).
PNG
media_image1.png
79
482
media_image1.png
Greyscale
In (1), nM and nN are the number of elements in M and N. Also Mk and Nk refer to cumulative probability distributions of M and N in kth bin, respectively. In each statistical significance level of α, if KS (M,N) < λα, the two random variables M and N are related to a unique distribution.”) [Examiner’s note: the “plurality of datasets” is being interpreted as the “cover” dataset and the “stegos” dataset. Because the features in these datasets are used to train a classifier to “predict suspect objects”, they are being interpreted as the “predictive features”. The “plurality of distribution tests” is being interpreted as the “cumulative probability distributions” and the “Kolmogorov-Smirnov statistical test”]
each dataset of the plurality of datasets comprises: (i) a set of training entries corresponding to a single predictive feature of the plurality of predictive features, (ii) a set of testing entries corresponding to the single predictive feature, and (Hosseini, Page 3, Col. 2, Section III, ¶[1-2]: “Quantity of 8,000 images were used as the cover images of training set and 2,000 remaining images were used as the cover images of testing set… In the first experiment, all the images in both training and testing sets were embedded by our LSB Matching simulator, with relative payload of 0.3bpp (bit per pixel). Hence, the training set included 8,000 covers and 8,000 stegos and the testing set included 2,000 covers and 2,000 stegos. Afterwards, 686 features of SPAM [3] were extracted from the covers and stegos of both sets, specially designed for detecting steganography in the spatial domain.”)
identifying, by one or more processors, a lead feature from the consistent feature set; (Hosseini, Page 2, Col. 1, ¶[3]: “in the proposed method, F is used just for ranking features. Afterwards, a strategy based on Kolmogorov-Smirnov (KS) test [16] is used in order to remove redundant features.”, Page 3, Col. 1, Section. B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
)[Examiner’s note: Fig. 2 discloses the features are ranked based on F statistical measure in descending order, with the most important feature placed first, which aligns with the concept of identifying the lead feature.]
selecting, by one or more processors, one of the features in the differential feature set; (Hosseini, Page 3, Col. 1, Section B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them.”, Page 3, Col. 1, Section B, ¶[4]: “The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”)
in response to determining that the correlation value is outside the correlation range, adjusting , by one or more processors, the differential feature set by removing the selected differential feature from the differential feature set; and (Hosseini, Page 3, Col. 1, Col. 1, Section B, ¶[3]: “Removing redundant features during forward comparison leads to sensitiveness of the algorithm to the ordering of the input features. To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section. In KS-CBF1 algorithm [19] which also uses KS for removing redundant features and is similar to the suggested algorithm, the features are reordered using a ranking system based on SUC2.”, Col. 2, Fig. 2:
PNG
media_image6.png
327
599
media_image6.png
Greyscale
) [Examiner’s note: The highlight indicates that the features that pass the KS test (i.e., have significantly different distribution) remain in the differential set, while features that fail the KS test (i.e., have nearly identical distribution to a higher-ranked feature) are removed. This means the algorithm in Fig. 2 is dynamically adjusting the differential set based on how similar each feature is to the lead feature]
generating, by one or more processors, a final feature set based on the adjusted differential feature set and the consistent feature set, wherein the final feature set, the consistent feature set, and the differential feature set are each generated independently of classification model execution. (Hosseini, Page 3, Col. 1, Section. B, ¶[1]: “The underlying idea of the proposed method is that some features have equal distributions and to form the final feature set, it is sufficient to select just one of them. In order to evaluate the similarity of the features, KS-test can be used and the evaluation can be made through a forward comparison”, Page 3, Col. 1, Section. B, ¶[3-4]: “To resolve this problem and make the results consistent, a sorting on the input features is required. In this paper it is suggested to sort features according to F statistic described in previous section… The proposed algorithm starts by sorting the features according to their F values in descending order; that is because the features having greater F can lead to more separability. In the next step, KS statistical measure is employed to find the features with equal distributions. In the cases where distributions are equal, the second feature is always discarded because it has a smaller F value.”) [Examiner’s note: Hosseini discloses evaluating the similarity of the features by using KS test then sorting them to form the optimal final feature set, which aligns with the concept of partitioning the features into differential set (e.g., set of features which do not share similarity) and consistent set (e.g., set of features which share similarity)]
building, by one or more processors, a machine learning model using the final feature set; (Hosseini, Pg. 1, Col. 1, Section I, ¶[1]: “The statistical measures which model these relations are known as features”, Col. 2, ¶[1]: “Kodovský [6] introduced rich models which were combination of smaller sub-models made to capture inter-block and intra-block relationships, form both the image and calibrated one. The final feature set included 22,510 features.”)
Hosseini fails to disclose:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features;
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry;
the set of testing entries are devoid of a target;
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries
partitioning, [by one or more processors], the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on [[their]] the corresponding distribution test, wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries
computing, [by one or more processors], a correlation range of the set of training entities based on a correlation between data values of the set of training entities and the lead feature;
computing, [by one or more processors], a correlation value of the set of testing entities based on a correlation between data values of the set of testing entities and the lead feature;
However, Pan explicitly discloses:
each distribution test of the plurality of distribution tests corresponds to a different predictive feature of a plurality of predictive features; (Pan, Abstract: “With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data.”, Pg. 2, Figure 2:
PNG
media_image2.png
295
572
media_image2.png
Greyscale
, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset. Classification performance better than random guess indicates the different feature distributions between the training and test datasets.”)
(iii) for each individual entry, of the set of training entries and the set of testing entries, a variable identifying a data source of the respective individual entry; (Pan, Pg. 2, Col. 1, ¶[1]: “In adversarial validation, a binary classifier, adversarial classifier, is trained to predict if a sample belongs to the test dataset”, Pg. 3, Col. 1, ¶1]: “We start with a labeled training dataset {(ytrain, Xtrain)} ∈ R × Rd , and an unlabeled test dataset {Xtest } ∈ Rd with an unknown conditional probability Py |X . Then, we train an adversarial classifier that predicts P({train, test }|X) to separate train and test, and generate the propensity score ppropensity = P(test |X) on both Xtest and Xtrain.”) [Examiner’s note: Pan’s adversarial classifier is trained to predict whether a sample is from training or testing dataset, which is effectively a source label.]
and each distribution test tests whether a value distribution of the set of training entries is different from that of the set of testing entries (Pan, Pg. 3, Col. 1, ¶[2]: “The feature importance and propensity score from the adversarial classifier can be used to detect concept drift between the training and test data, and provide insights on the cause of the concept drift such as which features and subsamples in the training data are most different from ones in the test data.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Pan. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Pan teaches an adversarial validation approach to concept drift problems in user targeting automation systems. One of ordinary skill would have motivation to combine Hosseini and Pan in order to select, retain or remove features based not only on predictive usefulness but also on whether those features remain stable across datasets, thereby reducing selection bias, improving robustness to drift, and producing a model that generalizes better to unseen test data.
However, Rajagopal explicitly discloses:
and (iii) a data source variable indicating a type of each entry of the set of training entries and the set of testing entries; (Rajagopal, Pg. 19729, Table 2:
PNG
media_image3.png
338
553
media_image3.png
Greyscale
)
wherein the consistent feature set includes features that have statistical consistency between the set of training entries and the set of testing entries (Rajagopal, Pg. 19732, Col. 1, ¶[5]: “While building machine learning models, it often becomes imperative to compare the performance of classifiers and the best way to achieve this is to perform statistical significance tests… In this context, the null hypothesis (H0) suggests that there is no performance difference among classifiers whereas an alternate hypothesis (H1) indicates that at least one classifier performs differently. Suppose, ‘d’ refers to the number of datasets and ‘k’ signifies the number of classifiers, Friedman test statistic can be calculated as shown in equation (7)
PNG
media_image4.png
63
460
media_image4.png
Greyscale
”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Rajagopal. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Rajagopal teaches a meta-classification approach using decision jungle to perform both binary and multiclass classification. One of ordinary skill would have motivation to combine Hosseini and Rajagopal because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Hassani explicitly discloses:
partitioning, [by one or more processors], the plurality of predictive features into: (i) a differential feature set and (ii) a consistent feature set, based on [[their]] the corresponding distribution test (Hassani, Page 594, ¶[2]: “Next, we introduce the hypothesis which are relevant for the proposed KSPA test. Let us begin by presenting the hypothesis for the two-sided KS test. Let X and Y be two random variables with c.d.f.’s FX and FY , respectively. Then, a two sample, two-sided KS test will test the hypothesis that both c.d.f.’s have an identical distribution, and the resulting null and alternate hypothesis can be expressed as:
PNG
media_image5.png
76
410
media_image5.png
Greyscale
In simple terms, the null hypothesis in Equation (6) states that both X and Y share an identical distribution whilst the alternate hypothesis states that X and Y do not share the same distribution.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Hassani. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Hassani teaches performing a similarity statistical test between two different datasets using Kolmogorov-Smirnov technique. One of ordinary skill would have motivation to combine Hosseini and Hassani in order to measure how different the distributions of a feature are across groups. If a feature behaves very differently between two classes, that’s a good sign it’s useful for making predictions, this is especially helpful in real-world situations where data doesn’t follow neat patterns – since KS doesn’t assume any particular distributions, it works well no matter how messy the data gets.
However, Haws explicitly discloses:
the set of testing entries are devoid of a target; (Haws, Col. 1, Lines 53-58: “The feature selection module is configured to perform a method. The method includes receiving a set of training samples and a set of test samples. The set of training samples includes a first set of features and a class value. The set of test samples includes the set of features absent the class value.”) [Examiner’s note: the testing dataset is devoid of the target e.g., the set of test samples includes the set of features absent the class value]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Haws. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Haws teaches a transductive feature selection method with maximum-relevancy and minimum-redundancy criteria. One of ordinary skill would have motivation to combine Hosseini and Haws because by excluding the target outcome from the testing dataset during feature selection process, the features are chosen based purely on the underlying data patterns – not on knowledge of the final outcome. This leads to a more honest evaluation of the model’s true predictive power and helps create models that are reliable, and robust when applied to unseen data
However, Yu explicitly discloses:
computing, [by one or more processors], a correlation range of the set of training entities based on a correlation between data values of the set of training entities and the lead feature; (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: the “lead feature” is being interpreted as the predominant feature Fi, and the “set of training entities” is being interpreted as “class C” because class C represents the target values that we are trying to predict]
computing, [by one or more processors], a correlation value of the set of testing entities based on a correlation between data values of the set of testing entities and the lead feature; (Yu, Page 4, Col. 1, Section 4.1, ¶[2]: “More specifically, suppose a data set S contains N features and a class C. Let SUi,c denote the SU value that measures the correlation between a feature Fi and the class C (named C-correlation), then a subset S’ of relevant features can be decided by a threshold SU value ᵹ, such that ⱯFi
∈
S’, 1
≤
I
≤
N, SUi,c
≥
ᵹ”, Page 4, Col. 2, ¶[2]: “The correlation between a feature Fi (Fi
∈
S) and the class C is predominant iff SUi,c
≥
ᵹ, and ⱯFj ϵ S’ (j
≠
i), there exists no Fj such that SUi,j
≥
SUi,c. If there exists such Fj to a feature Fi, we call it a redundant peer to Fi and use SPi to denote the set of all redundant peers for Fi. Given Fi
∈
S’ and SPi (SPi
≠
∅
), we divide SPi into two parts, SPi+ and SPi-, where SPi+ = {Fj|Fj
∈
SPi, SUj,c > SUi,c} and SPi- = {Fj|Fj
∈
SPi, SUj,c
≤
SUi,c}”) [Examiner’s note: the “lead feature” is being interpreted as the predominant feature Fi, and the “set of testing entities” is being interpreted as “class C” because class C represents the target values that we are trying to predict]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Hosseini and Yu. Hosseini teaches a new feature selection algorithm which utilizes two statistical measures (i.e., KS from Kolmogorov-Smirnov test and F from F-to-remove). Yu teaches a correlation-based feature selection method. One of ordinary skill would have motivation to combine Hosseini and Yu in order to identify redundancy and keep the model focused on the most valuable information. If two features are highly correlated, they are basically telling the model the same thing. Including both doesn’t add value – it just clutters the model, increases complexity, and can even lead to overfitting. By computing the correlation range, you can spots these overlaps early and remove redundant features.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY TRAN whose telephone number is (571)270-0693. The examiner can normally be reached Monday - Friday 7:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY TRAN/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights