DETAILED ACTION
The applicant’s amendment filed on March 30, 2026, has been acknowledged. Claims 1-20, as amended, are currently pending and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim recites the actions related to validating a model, which is a mental process.
Under Step 1, claims 1-16 recite a method or process, claims 17-18, recite a medium and claims 19-20 recite an apparatus. As such each of the claims falls within one of the statutory categories.
Under Step 2(a) – Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III).
In the instant case the method limitations require training a machine learning model and validating the machine learning model. Each of which is part of the mental process. The medium and apparatus require, receiving a training set, operating the machine learning model, comparing labeled and unlabeled objects, extracting a feature vector for an object and calculating a distance from the extracted feature vector to feature vector cluster. The last step includes comparing the computed classes to the estimated labels. Each of which is part of the mental process.
Further the specific manner of training the model and validating the data is not claimed and as such recites a general or generic action of training the model and validating. The remaining elements describe the type of data. That is describing what the training set comprises. The manner in which the classification is performed is also generic, the limitations establish it is based on computing a smallest distance but provides no specifics as to how the function is performed. The medium and the apparatus also recite generic functions such as receiving training set, operating the model, comparing the labeled and unlabeled data, extracting a feature vector, calculating the distance and comparing the computed classes are all generic. As such these elements are directed toward the abstract idea and merely applying the abstract idea on a computer, see MPEP 2106.05(f). While the claims recite elements such as computers, medium, processor and memory the claims still allows a person to review the data, classify and validate the results. As such the limitation amounts to merely applying the abstract idea to a computer, which is not enough to render the claims into a practical application as shown in MPEP 2106.05(f). As such when considered individually or in combination the elements fail to render the claims into a practical application.
Further the limitations themselves are considered to be merely applying the abstract idea of evaluating the performance of a model on a computer, as the claims merely recite an outcome and fail to recite any specific details as to how the solution was accomplished, as established in MPEP 2106.05(f). That is the claims recite various types of data which are considered but no specific manner of combining that data to achieve the result. While the claims reference that this is a form of validating the model for classifying malware the limitations fail to recite or establish any specific manner of validating as such this is not considered an improvement.
The claims have been amended to recite “extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values” which recite a function of extracting the values without any specifics as to how the function is performed. That is, it establishes that vector comprises a numerical value but not how the value is extracted. As such these elements are directed toward the abstract idea and merely applying the abstract idea on a computer, see MPEP 2106.05(f).
Step 2(a)(II) considers the additional elements of the independent claims with respect to transforming the abstract idea into a practical application. As noted the above the determination is generic and as such cannot be considered to be a practical application. The other steps of the independent claims amount to merely receiving data, and using that received data in any way to classify the data and validate the results, which again does not amount to be a practical application.
As state above the judicial exception is not integrated into a practical application. In particular, the claim recites additional elements – a processor, a memory, instructions and non-transitory computer readable medium. The hardware in claimed limitations is recited at a high-level of generality (i.e., as a generic component performing a generic functions ) such that it amounts no more than mere instructions to apply the exception using a generic components. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a processor and memory to receive, operate, compare, extract and compares the computed values amounts to no more than mere instructions to apply the exception using a generic component. Mere instructions to apply an exception using a generic components cannot provide an inventive concept. The claim is not patent eligible.
Step 2(b) considers the additional elements of the independent claims with respect to being significantly more than the identified abstract idea. As noted above there are no additional elements which indicate that the claims amount to significantly more than the abstract idea.
Claims 2, 18 and 20 recite “wherein the binary classifier is a malware classifier, and wherein the first class is a clean class the second class is a malware class” which describes the title of the classification but does not establish how the classification is performed. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 3 recites “assigning a confidence to the estimated classification” which describes assigning a value to the classification but does not establish how the confidence is calculated or assigned. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 4 recites “wherein the confidence comprises an absolute value difference between a distance to a nearest feature vector cluster for the first class and a distance to a nearest feature vector cluster for the second class”, which describes the confidence value but this again is merely a calculation which is part of the abstract idea and is not used as part of the classification. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 5, recites “assigning the unlabeled objects to confidence buckets based on the respective confidences of the unlabeled objects” which recites the function assigning objects to buckets based on the confidences but this is generic and not specific to how it is performed or how this is used to validate the data. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 6 recites “computing error bars for the confidence buckets” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 7 recites “identifying an outlier cluster of feature vectors outside of the error bars, and designating the outlier clusters for additional analysis” which recites the function but no specifics as to how it is achieved. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 8 recites “ aggregating counts of the first class and second class by confidence bucket” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 9 recites “calculating an error for the confidence bucket using an average prediction score and total count of feature vectors” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 10 recites “computing the respective known feature vector cluster” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 11 recites “computing the known feature vector cluster comprises receiving a set of feature vectors for known objects, including objects of the first class and objects of the second class, and computing for respective known feature vectors a smallest minimum distance to a cluster core” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 12 recites “computing the smallest minimum distance comprises generating a distance score array, calculating a sum of closest distances for individual feature vectors, and adding the sum to the distance score array” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 13 recites “wherein generating the distance score array comprises creating a zeroed array of distances with indexes matching an array of feature vectors, and for each of a set of input feature vectors, computing a distance between the input feature vector and an iterated feature vector” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 14 recites “wherein computing the smallest distance to respective known feature vector clusters comprises computing a first distance to a nearest cluster core of the first class and second distance to a nearest cluster core of the second class, and selecting an estimated class based on a lesser of the first distance and the second distance” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 15 recites “computing a model score for the ML model as disproportional to a ratio of total error amount to total number of model classifications” which recites a generic computation which is part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Claim 16 recites “plotting a model validation for the model and presenting the plotted model validation to a human user” which recites a part of the abstract idea. As such this amounts to merely applying the abstract idea on a computer, see MPEP 2106.05(f). As such this does not render the abstract idea into a practical application.
Thus, when considered individually or as a combination these elements do not amount to a practical application.
As such claims 1-20 recite an abstract idea and without any specifics to how the functions are performed the claims are not found to render the abstract idea into a practical application. Therefore, the claims have been rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 10-12, 14 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartos et al. (US 2019/0260775 A1) hereafter Bartos, in view of Chang et al. (US 8,442,321 B1) hereafter Chang, further in view of Klaiman et al. (WO 2021/099584 A1) hereafter Klaiman.
As per claim 1, Bartos discloses a computer-implemented method of validating a machine learning (ML) model (Bartos paragraph [0032]; discloses a machine learning model which utilizes clustering based on a notion of similarity. Paragraph [0042]; discloses that the machine learning models are trained using labeled data. Paragraph [0050]; discloses that the model is validated or refined removing any flaws or artifacts thereby increasing the efficacy), wherein the ML model is a binary classifier (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious.), the method comprising:
training the ML model on a training set comprising labeled objects from a first class and a second class, to provide a trained ML model (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious. Paragraph [0042]; discloses that the model is trained using labeled data); and
validating the ML model on a training set (Bartos Paragraph [0050]; discloses that the model is validated or refined removing any flaws or artifacts thereby increasing the efficacy), wherein the training set comprises at least some unlabeled objects, (Bartos paragraph [0053]; discloses that the training set includes unlabeled objects which are checked for similarities with candidates. Paragraph [0046]; discloses that the techniques use already labeled sampled to generate unknown labels).
While Bartos discusses finding similarities for unlabeled objects it is not explicit that it uses an estimated classification as a proxy for a known label, wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. Additionally, the reference fails to explicitly disclose extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values.
Chang, which like Bartos talks about utilizing machine learning on unlabeled data, teaches it is known to uses an estimated classification as a proxy for a known label, wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class (Chang Col. 8, line 55 through Col. 9, line 3; teaches that it is known when determining the similarity between classes it is known to determine the closest or smallest distance to the feature vector clusters for each class. Col. 5, line 62 through Col. 6, line 12; teaches that the similarity can be measured using Euclidean distance. Col. 6, lines 39-42; teaches that the similarity is minimizing distance which is the shortest or smallest distance. Col. 13, lines 31-44; teaches that the information is clustering from a center. Since Bartos already determines the similarity between the normal and malicious classifications it would have been obvious to classify the data based on which feature vector cluster it is closest to as shown in Chang. Chang establishes that these are known methods of determining similarity and would have been obvious to use to classify data).
Bartos discloses a system for training and validating a machine learning model which classifies malware. While Bartos establishes determining the similarity to known labels it is not explicit that the similarity is determined by estimating the classification as a proxy for a known label, wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class.
Chang, teaches that when determining the similarity of unlabeled data it is known to estimate the classification as a proxy for a known label, wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos the ability to determine the similarity by estimating the classification based on computing a smallest distance to respective known feature vector clusters as taught by Chang since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Chang, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, with the ability to determine the similarity by estimating the classification based on computing a smallest distance to respective known feature vector clusters as taught by Chang, for the purposes of refining the data over time and adding new data as it becomes available. Since Bartos already determines the similarity between the normal and malicious classifications it would have been obvious to classify the data based on which feature vector cluster it is closest to as shown in Chang.
The combination fails to explicitly disclose extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values.
Klaiman, which like the combination talks about feature vectors, teaches it is known to extract, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values (Page 34, lines 1-5, 14-24 and 32-33; teaches it is known to extract features from a training data set to create a feature vector. This is done to train different patterns of data. Page 36, lines 1-28; teaches calculating a numerical value which can be used to determine the similarity between patterns and apply labels. Page 40, lines 3-7; teaches that the distance can be measured between the centers or alternatively by measuring edges. Similar to what is shown in Chang. Since the combination already establishes validating the model on the training set where the training set includes an estimated classification based on smallest distance to known feature vectors, it would have been obvious to first extract the feature vector from the training set as shown in as this is a known technique to determine if the data is similar to established patterns to apply labels to data as shown in Klaiman).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang, teaches that when determining the similarity of unlabeled data it is known to estimate the classification as a proxy for a known label, wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. The combination fails to explicitly disclose extracting feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values.
Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. Klaiman establishes this type of extraction and comparison was known in the prior art at the time of the invention.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos and Chang the ability to d extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values as taught by Klaiman since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Klaiman, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos and Chang, with the ability to d extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values as taught by Klaiman, for the purposes of determining the similarity between patterns for labeling data. Since the combination already establishes validating the model on the training set where the training set includes an estimated classification based on smallest distance to known feature vectors, it would have been obvious to first extract the feature vector from the training set as shown in as this is a known technique to determine if the data is similar to established patterns to apply labels to data as shown in Klaiman.
As per claim 2, the combination of Bartos, Chang and Klaiman teaches the method of claim 1, Bartos further discloses wherein the binary classifier is a malware classifier, and wherein the first class is a clean class and the second class is a malware class (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious).
As per claim 10, the combination of Bartos, Chang and Klaiman teaches the method of claim 1, Bartos further discloses further comprising computing the respective known feature vector clusters (Bartos paragraphs [0057] and [0058]; discloses computing the feature vectors. Paragraph [0032]; discloses that clustering is used to identify anomalies).
As per claim 11, the combination of Bartos, Chang and Klaiman teaches the method of claim 10, Chang further teaches wherein computing the known feature vector cluster comprises receiving a set of feature vectors for known objects, including objects of the first class and objects of the second class, and computing for respective known feature vectors a smallest minimum distance to a cluster core (Chang Col. 8, line 55 through Col. 9, line 3; teaches that it is known when determining the similarity between classes it is known to determine the closest or smallest distance to the feature vector clusters for each class).
As per claim 12, the combination of Bartos, Chang and Klaiman teaches the method of claim 11, Chang further teaches wherein computing the smallest minimum distance comprises generating a distance score array, calculating a sum of closest distances for individual feature vectors, and adding the sum to the distance score array (Chang Col. 8, line 26 through Col. 9, line 3; teaches that the smallest or closet distance is generated by calculating the sum of the closest distances for the feature vectors, as shown in Col. 12, lines 29-44).
As per claim 14, the combination of Bartos, Chang and Klaiman teaches the method of claim 1, Chang further teaches wherein computing the smallest distance to respective known feature vector cluster comprises computing a first distance to a nearest cluster core of the first class and second distance to a nearest cluster core of the second class, and selecting an estimated class based on a lesser of the first distance or the second distance (Chang Col. 8, line 26 through Col. 9, line 3; teaches that the distance to each feature vector cluster is calculated and the nearest or closest is selected).
As per claim 17, Bartos discloses one or more tangible, non-transitory computer-readable media having stored thereon machine-executable instructions to validate a trained machine learning (ML) model (Bartos paragraph [0032]; discloses a machine learning model which utilizes clustering based on a notion of similarity. Paragraph [0042]; discloses that the machine learning models are trained using labeled data. Paragraph [0050]; discloses that the model is validated or refined removing any flaws or artifacts thereby increasing the efficacy. Paragraph [0146]; discloses that the instructions are stored on a non-transitory computer readable medium. ), wherein the ML model is a binary classifier (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious.), and wherein validating the ML model comprises:
receiving a training set, the training set comprising objects including both labeled objects and unlabeled objects (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious. Paragraph [0042]; discloses that the model is trained using labeled data. Paragraph [0046]; discloses that it includes unlabeled data);
operating the ML model to classify the objects of the training set as belonging to a first class or a second class according to computed classifications (Bartos Paragraph [0044]; discloses that the machine learning model is used to classify objects specifically compared to known samples to determine if it is malicious or not. Thus the elements are classified as normal or malicious. Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious);
for labeled objects of the training set, comparing the computed classification to known classes of the labeled objects (Bartos paragraph [0054]-[0055]; discloses that labeled data can be successfully be detected and used to classify objects), and
for unlabeled objects, comparing the computed classifications to estimated labels, comprising estimating labels for at least some of the unlabeled objects (Bartos paragraph [0053]; discloses that the training set includes unlabeled objects which are checked for similarities with candidates),
While Bartos discusses finding similarities for unlabeled objects it is not explicit that it wherein estimating labels comprises extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class. Additionally, the reference fails to explicitly disclose extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values.
Chang, which like Bartos talks about utilizing machine learning on unlabeled data, teaches it is known wherein estimating labels comprises extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class (Chang Col. 8, line 55 through Col. 9, line 3; teaches that it is known when determining the similarity between classes it is known to determine the closest or smallest distance to the feature vector clusters for each class. Col. 5, line 62 through Col. 6, line 12; teaches that the similarity can be measured using Euclidean distance. Col. 6, lines 39-42; teaches that the similarity is minimizing distance which is the shortest or smallest distance. Col. 13, lines 31-44; teaches that the information is clustering from a center. Since Bartos already determines the similarity between the normal and malicious classifications it would have been obvious to classify the data based on which feature vector cluster it is closest to as shown in Chang. Chang establishes that these are known methods of determining similarity and would have been obvious to use to classify data).
Bartos discloses a system for training and validating a machine learning model which classifies malware. While Bartos establishes determining the similarity to known labels it is not explicit that the similarity is determined by extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class.
Chang, teaches that when determining the similarity of unlabeled data it is known to extract a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos the ability to determine the similarity by extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class as taught by Chang since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Chang, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, with the ability to determine the similarity by extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class as taught by Chang, for the purposes of refining the data over time and adding new data as it becomes available. Since Bartos already determines the similarity between the normal and malicious classifications it would have been obvious to classify the data based on which feature vector cluster it is closest to as shown in Chang.
The combination fails to explicitly disclose extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values.
Klaiman, which like the combination talks about feature vectors, teaches it is known to extract, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values (Page 34, lines 1-5, 14-24 and 32-33; teaches it is known to extract features from a training data set to create a feature vector. This is done to train different patterns of data. Page 36, lines 1-28; teaches calculating a numerical value which can be used to determine the similarity between patterns and apply labels. Page 40, lines 3-7; teaches that the distance can be measured between the centers or alternatively by measuring edges. Similar to what is shown in Chang. Since the combination already establishes validating the model on the training set where the training set includes an estimated classification based on smallest distance to known feature vectors, it would have been obvious to first extract the feature vector from the training set as shown in as this is a known technique to determine if the data is similar to established patterns to apply labels to data as shown in Klaiman).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang, teaches that when determining the similarity of unlabeled data it is known to estimate the classification as a proxy for a known label, wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. The combination fails to explicitly disclose extracting feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values.
Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. Klaiman establishes this type of extraction and comparison was known in the prior art at the time of the invention.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos and Chang the ability to d extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values as taught by Klaiman since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Klaiman, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos and Chang, with the ability to d extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values as taught by Klaiman, for the purposes of determining the similarity between patterns for labeling data. Since the combination already establishes validating the model on the training set where the training set includes an estimated classification based on smallest distance to known feature vectors, it would have been obvious to first extract the feature vector from the training set as shown in as this is a known technique to determine if the data is similar to established patterns to apply labels to data as shown in Klaiman.
As per claim 18, the combination of Bartos, Chang and Klaiman teaches the one or more tangible, non-transitory computer-readable media of claim 17, Bartos further discloses wherein the binary classifier is a malware classifier, and wherein the first class is a clean class the second class is a malware class (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious).
As per claim 19, Bartos discloses a computing apparatus, comprising: a processor circuit and a memory; and instructions encoded within the memory to instruct the processor circuit to validate a trained machine learning (ML) model (Bartos paragraph [0032]; discloses a machine learning model which utilizes clustering based on a notion of similarity. Paragraph [0042]; discloses that the machine learning models are trained using labeled data. Paragraph [0050]; discloses that the model is validated or refined removing any flaws or artifacts thereby increasing the efficacy. Paragraph [0146]; discloses that the instructions are stored on a non-transitory computer readable medium and executed by a computer which includes both a processor and memory), wherein the ML model is a binary classifier to classify objects into a first class and a second class (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious), and wherein validating the ML model comprises:
receiving a training set, the training set including both labeled objects and unlabeled objects (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious. Paragraph [0042]; discloses that the model is trained using labeled data. Paragraph [0046]; discloses that it includes unlabeled data);
operating the ML model to classify objects of the training set and provide computed classes for the objects (Bartos Paragraph [0044]; discloses that the machine learning model is used to classify objects specifically compared to known samples to determine if it is malicious or not. Thus, the elements are classified as normal or malicious);
for labeled objects, comparing the computed classes to the known class of the labeled objects (Bartos paragraph [0054]-[0055]; discloses that labeled data can be successfully be detected and used to classify objects);
for unlabeled objects, providing estimated labels (Bartos paragraph [0053]; discloses that the training set includes unlabeled objects which are checked for similarities with candidates),
and comparing the computed classes to the estimated labels (Bartos paragraph [0132]-[0133]; discloses the computed values are compared to the estimated buckets to refine the values over time and remove false positives).
While Bartos discusses finding similarities for unlabeled objects it is not explicit that it wherein estimating labels comprises extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class. Additionally, the reference fails to explicitly disclose extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values.
Chang, which like Bartos talks about utilizing machine learning on unlabeled data, teaches it is known wherein estimating labels comprises extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class (Chang Col. 8, line 55 through Col. 9, line 3; teaches that it is known when determining the similarity between classes it is known to determine the closest or smallest distance to the feature vector clusters for each class. Col. 5, line 62 through Col. 6, line 12; teaches that the similarity can be measured using Euclidean distance. Col. 6, lines 39-42; teaches that the similarity is minimizing distance which is the shortest or smallest distance. Since Bartos already determines the similarity between the normal and malicious classifications it would have been obvious to classify the data based on which feature vector cluster it is closest to as shown in Chang. Chang establishes that these are known methods of determining similarity and would have been obvious to use to classify data).
Bartos discloses a system for training and validating a machine learning model which classifies malware. While Bartos establishes determining the similarity to known labels it is not explicit that the similarity is determined by extracting a feature vector for an object and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class.
Chang, teaches that when determining the similarity of unlabeled data it is known to extract a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos the ability to determine the similarity by extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class as taught by Chang since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Chang, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, with the ability to determine the similarity by extracting a feature vector for an object, and calculating a distance from the extracted feature vector to feature vector clusters for the first class and second class as taught by Chang, for the purposes of refining the data over time and adding new data as it becomes available. Since Bartos already determines the similarity between the normal and malicious classifications it would have been obvious to classify the data based on which feature vector cluster it is closest to as shown in Chang.
The combination fails to explicitly disclose extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values.
Klaiman, which like the combination talks about feature vectors, teaches it is known to extract, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values (Page 34, lines 1-5, 14-24 and 32-33; teaches it is known to extract features from a training data set to create a feature vector. This is done to train different patterns of data. Page 36, lines 1-28; teaches calculating a numerical value which can be used to determine the similarity between patterns and apply labels. Page 40, lines 3-7; teaches that the distance can be measured between the centers or alternatively by measuring edges. Similar to what is shown in Chang. Since the combination already establishes validating the model on the training set where the training set includes an estimated classification based on smallest distance to known feature vectors, it would have been obvious to first extract the feature vector from the training set as shown in as this is a known technique to determine if the data is similar to established patterns to apply labels to data as shown in Klaiman).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang, teaches that when determining the similarity of unlabeled data it is known to estimate the classification as a proxy for a known label, wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. The combination fails to explicitly disclose extracting feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values.
Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. Klaiman establishes this type of extraction and comparison was known in the prior art at the time of the invention.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos and Chang the ability to d extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values as taught by Klaiman since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Klaiman, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos and Chang, with the ability to d extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values as taught by Klaiman, for the purposes of determining the similarity between patterns for labeling data. Since the combination already establishes validating the model on the training set where the training set includes an estimated classification based on smallest distance to known feature vectors, it would have been obvious to first extract the feature vector from the training set as shown in as this is a known technique to determine if the data is similar to established patterns to apply labels to data as shown in Klaiman.
As per claim 20, the combination of Bartos, Chang and Klaiman teaches the computing apparatus of claim 19, Bartos further discloses wherein the binary classifier is a malware classifier, and wherein the first class is a clean class the second class is a malware class (Bartos Paragraph [0037]; discloses that the classifications for the data is binary that is a first and second class. In this case the two classes are normal and malicious).
Claim(s) 3-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartos et al. (US 2019/0260775 A1) hereafter Bartos, in view of Chang et al. (US 8,442,321 B1) hereafter Chang, further in view of Klaiman et al. (WO 2021/099584 A1) hereafter Klaiman, further in view of Dennison et al. (US 9,558,352 B1) hereafter Dennison.
As per claim 3, the combination of Bartos, Chang and Klaiman teaches the method of claim 1, however the combination is not explicit that it further discloses further comprising assigning a confidence to the estimated classification.
Dennison, which like the combination talks about using training a model, teaches it is known to assign a confidence to the estimated classification (Dennison Col. 22, line 55 through Col. 23, line 10; teaches it is known to assign a confidence or likelihood to the estimated classification. This establishes the likelihood that the system sees the estimation as correct. Since the combination already determines the classification based on the distance to the nearest vector cluster, it would have been obvious to assign a confidence score as well to establish how likely the classification is correct as shown in Dennison).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. While the combination establish determining the closest feature vector cluster, it is not explicit that it assigns a confidence to the estimated classification.
Dennison, teaches it is known to assign a confidence to the estimated classification.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos, Chang and Klaiman the ability to assign a confidence to the estimated classification as taught by Dennison since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Dennison, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, Chang and Klaiman, with the ability to assign a confidence to the estimated classification as taught by Dennison, for the purposes of indicating the likelihood the classification is correct. Since the combination already determines the classification based on the distance to the nearest vector cluster, it would have been obvious to assign a confidence score as well to establish how likely the classification is correct as shown in Dennison.
As per claim 4, the combination of Bartos, Chang, Klaiman and Dennison teaches the method of claim 3, Chang further discloses wherein the confidence comprises an absolute difference between a distance to a nearest feature vector cluster for the first class and a distance to a nearest feature vector cluster for the second class (Chang Col. 8, line 55 through Col. 9, line 3; teaches that it is known when determining the similarity between classes it is known to determine the closest or smallest distance to the feature vector clusters for each class. The distance to each feature vector cluster is computed and used to determine the most likely match. Col. 12, lines 29-44; teaches that this value can be refined over time as part of the feedback process).
As per claim 5, the combination of Bartos, Chang, Klaiman and Dennison teaches the method of claim 3, Bartos further discloses further comprising assigning the unlabeled objects to confidence buckets based on the respective confidences of the unlabeled objects (Bartos paragraph [0057]; discloses that unlabeled objects are assigned buckets based on their similarity).
Claim(s) 6-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartos et al. (US 2019/0260775 A1) hereafter Bartos, in view of Chang et al. (US 8,442,321 B1) hereafter Chang, further in view of Klaiman et al. (WO 2021/099584 A1) hereafter Klaiman, further in view of Dennison et al. (US 9,558,352 B1) hereafter Dennison, further in Jiang et al. (US 2022/0111839 A1) hereafter Jiang.
As per claim 6, the combination of Bartos, Chang, Klaiman and Dennison teaches the method of claim 5, however the combination fails to further discloses further comprising computing error bars for the confidence buckets.
Jiang, which like the combination talks about bucketing data, teaches it is known to comprise computing error bars for the confidence buckets (Jiang paragraph [0044]; teaches it is known to calculate and display error bars as a way to indicate quality level. Since the combination already indicates the confidence level it would have been obvious to indicate these levels as error bars as shown explicitly in Jiang as this is one known method of indicating the data).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. Dennison, teaches it is known to assign a confidence to the estimated classification. While the combination establishes indicating a confidence level it is not explicit that it computes error bars for the confidence buckets.
Jiang, teaches it is known to comprise computing error bars for the confidence buckets.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos, Chang, Klaiman and Dennison the ability to compute error bars for the confidence buckets as taught by Jiang since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Jiang, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, Chang, Klaiman and Dennison, with the ability to compute error bars for the confidence buckets as taught by Jiang, for the purposes of indicating the likelihood the classification is correct. Since the combination already indicates the confidence level it would have been obvious to indicate these levels as error bars as shown explicitly in Jiang as this is one known method of indicating the data.
As per claim 7, the combination of Bartos, Chang, Klaiman, Dennison and Jiang teaches the method of claim 6, Bartos further discloses further comprising identifying an outlier cluster of feature vectors outside of the error bars, and designating the outlier cluster for additional analysis (Bartos paragraph [0032]; discloses that the calculation includes identifier an outlier for the cluster of the feature vectors and using this information for additional analysis as part of the reflecting a degree of anomaly).
As per claim 8, the combination of Bartos, Chang, Klaiman and Dennison teaches the method of claim 5, however the combination fails to further discloses further comprising aggregating counts of the first class and second class by confidence bucket.
Jiang, which like the combination talks about bucketing data, teaches it is known to aggregate counts of the first class and second class by confidence bucket (Jiang paragraph [0044]; teaches that the data can be classified into confidence buckets based on the level of confidence. Since the combination already indicates the confidence level it would have been obvious to indicate these levels as confidence buckets as shown explicitly in Jiang as this is one known method of indicating the data).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. Dennison, teaches it is known to assign a confidence to the estimated classification. While the combination establishes indicating a confidence level it is not explicit that it aggregates counts of the first class and second class by confidence bucket.
Jiang, teaches it is known to comprise aggregate counts of the first class and second class by confidence bucket.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos, Chang, Klaiman and Dennison the ability to aggregate counts of the first class and second class by confidence bucket as taught by Jiang since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Jiang, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, Chang, Klaiman and Dennison, with the ability to aggregate counts of the first class and second class by confidence bucket as taught by Jiang, for the purposes of indicating the likelihood the classification is correct. Since the combination already indicates the confidence level it would have been obvious to indicate these levels as error bars as shown explicitly in Jiang as this is one known method of indicating the data.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartos et al. (US 2019/0260775 A1) hereafter Bartos, in view of Chang et al. (US 8,442,321 B1) hereafter Chang, further in view of Klaiman et al. (WO 2021/099584 A1) hereafter Klaiman, further in view of Dennison et al. (US 9,558,352 B1) hereafter Dennison, further in Jiang et al. (US 2022/0111839 A1) hereafter Jiang, Ramjeet et al. (EP 3266877 A1) hereafter Ramjeet.
As per claim 9, the combination of Bartos, Chang, Klaiman, Dennison and Jiang teaches the method of claim 8, the combination fails to further discloses further comprising calculating an error for the confidence bucket using an average prediction score and total count of feature vectors.
Ramjeet, which like the combination talks about analyzing the data, teaches it is known to calculate an error for the confidence bucket using an average prediction score and total count of feature vectors (Ramjeet paragraph [0072]; teaches that when calculating the error it is known to use an average prediction score and total count of feature vectors. Since the combination is determining the accuracy of the results it would have been obvious to calculate the error rate and the validity of the data using the techniques shown in Ramjeet).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. Dennison, teaches it is known to assign a confidence to the estimated classification. Jiang, teaches it is known to comprise computing error bars for the confidence buckets. While the combination establishes indicating an error or confidence level it is not explicit that it uses an average prediction score and total count of feature vectors.
Ramjeet, teaches it is known to calculate the error using an average prediction score and total count of feature vectors.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos, Chang, Dennison and Jiang the ability to calculate the error using an average prediction score and total count of feature vectors as taught by Ramjeet since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Ramjeet, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, Chang, Dennison and Jiang, with the ability to calculate the error using an average prediction score and total count of feature vectors as taught by Ramjeet, for the purposes of validating the data. Since the combination is determining the accuracy of the results it would have been obvious to calculate the error rate and the validity of the data using the techniques shown in Ramjeet.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartos et al. (US 2019/0260775 A1) hereafter Bartos, in view of Chang et al. (US 8,442,321 B1) hereafter Chang, further in view of Klaiman et al. (WO 2021/099584 A1) hereafter Klaiman, further in view of Grodek et al. (US 2016/0261743 A1) hereafter Grodek.
As per claim 13, the combination of Bartos, Chang and Klaiman teaches the method of claim 12, the combination fails to further discloses wherein generating the distance score array comprises creating a zeroed array of distances with indexes matching an array of feature vectors, and for each of a set of input feature vectors, computing a distance between the input feature vector and an iterated feature vector.
Grodek, which like the combination talks about calculating the distance between values, teaches generating the distance score array comprises creating a zeroed array of distances with indexes matching an array of feature vectors, and for each of a set of input feature vectors, computing a distance between the input feature vector and an iterated feature vector (Grodek paragraph [0041]; teaches it is a known approach to calculate the difference by first creating a zeroed array. Since Chang already calculates the distance between vectors to determine the shortest distance it would have been obvious to use the techniques shown in Grodek to achieve the result).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. While the combination establishes determining the distance between feature vectors it is not explicit that it creates a zeroed array of distances.
Grodek, teaches it is known to create a zeroed array of distances to determine the smallest distance.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos, Chang and Klaiman the ability to create a zeroed array of distances to determine the smallest distance as taught by Grodek since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Grodek, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, Chang and Klaiman, with the ability to create a zeroed array of distances to determine the smallest distance as taught by Grodek, for the purposes of checking distance. Since Chang already calculates the distance between vectors to determine the shortest distance it would have been obvious to use the techniques shown in Grodek to achieve the result.
Claim(s) 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bartos et al. (US 2019/0260775 A1) hereafter Bartos, in view of Chang et al. (US 8,442,321 B1) hereafter Chang, further in view of Klaiman et al. (WO 2021/099584 A1) hereafter Klaiman, further in view of Keyes (WO 01/50322 A2) hereafter Keyes.
As per claim 15, the combination of Bartos, Chang and Klaiman teaches the method of claim 1, however the combination fails to further discloses further comprising computing a model score for the ML model as disproportional to a ratio of total error amount to total number of model classifications.
Keyes, which like the combination talks about evaluating a model, teaches it is known to compute a model score for the ML model as disproportional to a ratio of total error amount to total number of model classifications (Keyes Page 30, lines 5-13; teaches that the lower the error rate the better the model. Since the combination is improving the values over time it would have been obvious for the value or score of the model to be disproportional to a ratio of total errors to total classifications or error rate as shown in Keyes the lower one is the higher the other is).
Bartos discloses a system for training and validating a machine learning model which classifies malware. Chang wherein the estimated classification is based on computing a smallest distance to respective known feature vector clusters for the first and second class. Klaiman, teaches that when determining the similarity between data it is known to extract feature vectors from objects of the training set and wherein each feature vector comprises a plurality of numerical feature values. While the combination establishes determining the flaws or problems in a model, it fails to explicit disclose computing a model score for the ML model as disproportional to a ratio of total error amount to total number of model classifications.
Keyes, teaches it is known to compute a model score for the ML model as disproportional to a ratio of total error amount to total number of model classifications.
It would have been obvious to one of ordinary skill in the art to include in the machine learning validation system of Bartos, Chang and Klaiman the ability to compute a model score for the ML model as disproportional to a ratio of total error amount to total number of model classifications as taught by Keyes since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Therefore, from this teaching of Keyes, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of validating machine learning models provided by Bartos, Chang and Klaiman, with the ability to compute a model score for the ML model as disproportional to a ratio of total error amount to total number of model classifications as taught by Keyes, for the purposes of indicating the accuracy of the model. Since the combination is improving the values over time it would have been obvious for the value or score of the model to be disproportional to a ratio of total errors to total classifications or error rate as shown in Keyes the lower one is the higher the other is.
As per claim 16, the combination of Bartos, Chang, Klaiman and Keyes teaches the method of claim 15, Bartos further discloses further comprising plotting a model validation for the model, and presenting the plotted model validation to a human user (Figures 4A-4D and paragraphs [0055]-[0057]; disclose that the model data is plotted and presented ).
Response to Arguments
Applicant's arguments filed March 30, 2026 have been fully considered but they are not persuasive.
In response to the applicant’s arguments on page 7, regarding the 101 rejections specifically that, “The Examiner rejects claims 1 - 20 under 35 USC§ 101 as being drawn to an abstract idea. Applicant respectfully traverses this rejection, and expressly denies that the claims, as previously presented, recite an abstract idea. Nevertheless, to expedite prosecution, the Applicant has made a modest amendment to the claims to address the Examiner's concern. The Applicant respectfully submits that the claims, as amended, are allowable under 35 USC § 101.”
The Examiner respectfully disagrees.
As stated above the claims have been amended to recite “extracting, by one or more processors, a plurality of feature vectors from objects of the training set, wherein each feature vector comprises a plurality of numerical feature values” which recite a function of extracting the values without any specifics as to how the function is performed. That is, it establishes that vector comprises a numerical value but not how the value is extracted. As such these elements are directed toward the abstract idea and merely applying the abstract idea on a computer, see MPEP 2106.05(f). Lacking any specifics to how the values are calculated and used the limitations continue to merely apply the abstract idea on a computer, therefore the rejections have been maintained.
In response to the applicant’s arguments on pages 7-9 regarding the art rejections, specifically that, “With respect to claim 1, the Examiner asserts that Bartos discloses a method of validating a machine learning model, and that Chang teaches "estimating the classification based on computing a smallest distance to respective known feature vector clusters for the first and second class." The Examiner further asserts that "Chang establishes that these are known methods of determining similarity and would have been obvious to use to classify data."”
“The Examiner's reliance on Chang is misplaced. Chang does not teach what the Examiner claims it teaches.”
“Chang relates to object recognition in images, specifically training a deep learning model to recognize objects using edge detection and image patches. Chang discloses a method of calculating "similarity distance" between detected edges in an image and pre-stored image patches in an image patch library. The similarity distance calculation in Chang is used to determine whether an image contains a particular object by matching detected edges to stored image patches.”
“Chang does not disclose or suggest "feature vector clusters" for first and second classes. Chang teaches an "image patch library" comprising pre-stored patches from training images, not feature vector clusters organized by class. The "similarity distance" in Chang is calculated between detected edges and image patches, not between feature vectors and cluster centroids or cluster cores.”
“The Examiner has not identified any disclosure in Chang that teaches clustering feature vectors into groups based on class labels, computing cluster cores or centroids, or determining a smallest distance to such clusters. Chang's "similarity distance" operates on a fundamentally different data structure-image edges matched against image patches-than the claimed "smallest distance to respective known feature vector clusters."” “Furthermore, Bartos does not cure this deficiency. The Examiner acknowledges that Bartos "discusses finding similarities for unlabeled objects" but is "not explicit" about using estimated classification based on smallest distance to feature vector clusters. The Examiner cites paragraph [0053] of Bartos, which describes "the refinement looks into the large set of unlabeled real data for similarities with the candidates."”
“Bartos paragraph [0053] teaches finding real-world samples that match synthetic malware candidates. This is fundamentally different from estimating a classification for unlabeled objects using distance to feature vector clusters. Bartos uses unlabeled data to refine synthetic samples for training-not to validate a model using estimated labels as proxies for known labels.”
“Bartos does not disclose estimating a label for an unlabeled object based on the object's distance to a feature vector cluster. The "similarity" discussed in Bartos paragraph [ 0053] is a matching operation between synthetic candidates and real samples, not a classification or estimation based on cluster distance.”
“The combination of Bartos and Chang fails to teach the claimed limitation because neither reference teaches or suggests computing a smallest distance to known feature vector clusters for the purpose of estimating classifications for unlabeled objects in a model validation context.”
“The other independent claims recite limitations similar, but not identical to, claim 1. These claims are allowable for analogous reasons, e.g., because the cited references do not teach the claim limitations. Furthermore, the dependent claims that depend from these claims are also allowable, as depending from an allowable base claim. The Applicant respectfully requests notice to this effect.”
The Examiner respectfully disagrees.
The applicant has alleged that the Chang reference merely establishes calculating similarity based on detected edges, while Chang establishes clustering values and determining the value from the center. Chang Col. 8, line 55 through Col. 9, line 3; teaches that it is known when determining the similarity between classes it is known to determine the closest or smallest distance to the feature vector clusters for each class. Col. 5, line 62 through Col. 6, line 12; teaches that the similarity can be measured using Euclidean distance. Col. 6, lines 39-42; teaches that the similarity is minimizing distance which is the shortest or smallest distance. Col. 13, lines 31-44; teaches that the information is clustering from a center. Since Bartos already determines the similarity between the normal and malicious classifications it would have been obvious to classify the data based on which feature vector cluster it is closest to as shown in Chang. Chang establishes that these are known methods of determining similarity and would have been obvious to use to classify data. This is further established by the Klaiman reference Page 40, lines 3-7; teaches that the distance can be measured between the centers or alternatively by measuring edges. As such the Examiner asserts that the combination does in fact read over the claims as written. Lacking any additional arguments the Examiner has not been persuaded and the rejections have therefore been maintained.
All rejections made towards the dependent claims are maintained due to the lack of a reply by the applicant in regards to distinctly and specifically point out the supposed errors in the Examiner’s action in the prior Office Action (37 CFR 1.111). The Examiner asserts that the applicant only argues that the dependent claims should be allowable because the independent claims are unobvious and patentable over Bartos in view of Chang, and, where appropriate, in further view of Dennison, Jiang, Ramjeet, Grodek and Keyes.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL R FISHER whose telephone number is (571)270-5097. The examiner can normally be reached Monday - Friday 9 am to 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Yin-Chen Shaw can be reached at (571)272-8878. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
PAUL R. FISHER
Primary Examiner
Art Unit 2498
/PAUL R FISHER/Primary Examiner, Art Unit 2498 10/7/2026
