DETAILED ACTION
Claims 1-20 are pending and have been examined.
--
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 6-7 and 13-16 are objected to because of the following informalities:
In claims 1, 15 and 16, “selecting, from the at least two putative values, the particular output threshold value… result in a maximal value of the at least one metric for the particular slice” should be “selecting, from the at least two putative values, the particular output threshold value… results in a maximal value of the at least one metric for the particular slice”
In claim 6, “selecting, from the at least two candidate values, the particular output threshold value… result in a maximal value of the at least one secondary metric for the particular slice” should be “selecting, from the at least two candidate values, the particular output threshold value… results in a maximal value of the at least one secondary metric for the particular slice”
In claim 7, “wherein determining which of the at least two putative values… result in a maximal value” should be “wherein determining which of the at least two putative values… results in a maximal value”
In claim 13, “selecting, from the at least two updated putative values, the updated output threshold value… result in a maximal value of the at least one metric for the particular slice” should be “selecting, from the at least two updated putative values, the updated output threshold value… results in a maximal value of the at least one metric for the particular slice”
In claim 14, “at least one input sample of the of the input data” should be “at least one input sample of the input data”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-4 are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 3 recites the limitation “the user.” There is insufficient antecedent basis for this limitation in the claim. For examination purposes examiner has interpreted “the user” to be “a user.”
Claim 4 is also rejected due to their dependency on a rejected claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
-
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more
Step 1: Claims 1-14 recite a method. Claim 15 recite a de-vice comprising processors and a non-transitory medium. Claims 16-20 recites an article of manufacture including a non-transitory computer-readable medium. Therefore, claims 1-14 are directed to a process, claim 15 is directed to a machine, and claims 16-20 are directed to a manufacture.
With respect to claims 1, 15 and 16:
2A Prong 1: The claim recites a judicial exception.
assigning each input sample of the input data to a respective slice of a plurality of slices (mental process – evaluation or judgement, a person can manually assign input sample to a slice)
determining a respective output threshold value for each slice in the plurality of slices, wherein determining a particular output threshold value for a particular slice in the plurality of slices comprises (mental process – evaluation or judgement, a person can manually determine a threshold value for each slice or for a particular slice)
determining at least two putative values of the particular output threshold value that, when applied to the plurality of model outputs corresponding to the particular slice, satisfy the at least one constraint for the particular slice (mental process – evaluation or judgement, a person can manually determine two putative values, when applying model outputs, that satisfy the constraint for the slice)
selecting, from the at least two putative values, the particular output threshold value for the particular slice by determining which of the at least two putative values, when applied to the plurality of model outputs corresponding to the particular slice, result in a maximal value of the at least one metric for the particular slice (mental process – evaluation or judgement, a person can manually select, from the two putative values, the particular threshold value, by determining which of the two putative values, results in a maximal value of the metric)
2A Prong 2: The judicial exception is not integrated into a practical application.
(claim 15) a controller comprising one or more processors; and a non-transitory computer readable medium having stored therein instructions executable by the controller device to cause the one or more processors to perform controller operations comprising (claim 16) including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising (mere instructions to apply an exception, (2) Whether the claim invokes computers - MPEP 2106.05(f); generic computer components)
obtaining input data, wherein the input data includes a plurality of input samples (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
for each slice in the plurality of slices, obtaining a respective at least one constraint and a respective at least one metric (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
obtaining a trained machine learning model (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
applying each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of model outputs corresponding to the particular slice (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying input sample to the trained model)
providing the respective output threshold value determined for each slice in the plurality of slices for application with the trained machine learning model (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
(claim 15) a controller comprising one or more processors; and a non-transitory computer readable medium having stored therein instructions executable by the controller device to cause the one or more processors to perform controller operations comprising (claim 16) including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising (mere instructions to apply an exception, (2) Whether the claim invokes computers - MPEP 2106.05(f); generic computer components)
obtaining input data, wherein the input data includes a plurality of input samples (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
for each slice in the plurality of slices, obtaining a respective at least one constraint and a respective at least one metric (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
obtaining a trained machine learning model (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
applying each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of model outputs corresponding to the particular slice (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying input sample to the trained model)
providing the respective output threshold value determined for each slice in the plurality of slices for application with the trained machine learning model (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claims 2 and 17:
2A Prong 1: The claim recites a judicial exception.
determining that the additional input sample corresponds to the particular slice (mental process – evaluation or judgement, a person can manually determine that the input sample corresponds to the particular slice)
responsive to determining that the additional input sample corresponds to the particular slice, applying the particular output threshold value to the additional model output to determine, for the additional input sample, a classification (mental process – evaluation or judgement, a person can manually apply the threshold value to the model output to determine a classification)
2A Prong 2: The judicial exception is not integrated into a practical application.
obtaining an additional input sample (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
applying the additional input sample to the trained machine learning model to generate an additional model output (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying the input sample to the trained model)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
obtaining an additional input sample (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
applying the additional input sample to the trained machine learning model to generate an additional model output (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying the input sample to the trained model)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claims 3 and 18:
2A Prong 1: The claim recites a judicial exception.
wherein the method further comprises: based on the classification determined for the additional input, accepting the input from the user as authentic (mental process – evaluation or judgement, a person can manually evaluate if the input is authentic based on the classification)
2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the input sample represents an input from the user, and (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting; claim 1 recites “obtaining input data, wherein the input data includes a plurality of input samples” which is insignificant extra-solution activity. Specifying the input sample representing an input from the user does not cause the limitation to integrate the exception into a practical application.)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
wherein the input sample represents an input from the user, and (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i); claim 1 recites “obtaining input data, wherein the input data includes a plurality of input samples” which is insignificant extra-solution activity. Specifying the input sample representing an input from the user does not cause the limitation to be significantly more than the judicial exception.)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 4:
2A Prong 2: The judicial exception is not integrated into a practical application.
wherein the input from the user represents an update to a map, and (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting; claim 3 recites “… an input from the user” which is insignificant extra-solution activity. Specifying the input representing an update to a map does not cause the limitation to integrate the exception into a practical application.)
wherein accepting the input from the user as authentic comprises updating a map database based on the update to the map (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
wherein the input from the user represents an update to a map, and (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i); claim 3 recites “… an input from the user” which is insignificant extra-solution activity. Specifying the input representing an update to a map does not cause the limitation to be significantly more than the judicial exception.)
wherein accepting the input from the user as authentic comprises updating a map database based on the update to the map (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: storing and retrieving information in memory, Versata Dev. Group, Inc. - MPEP 2106.05(d)(II)(iv))
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 5:
2A Prong 1: The claim recites a judicial exception.
wherein determining at least two putative values of the particular output threshold value that, when applied to the plurality of model outputs corresponding to the particular slice, satisfy the at least one constraint for the particular slice comprises: (mental process – evaluation or judgement, a person can manually determine two values that satisfy the constraint)
determining whether each possible threshold value of a discrete set of possible values of the particular output threshold value, when applied to the plurality of model outputs corresponding to the particular slice, satisfies the at least one constraint for the particular slice, wherein the discrete set of possible values of the particular output threshold value span a range of values (mental process – evaluation or judgement, a person can manually determine whether each possible value satisfies the constraint)
With respect to claim 6:
2A Prong 1: The claim recites a judicial exception.
wherein selecting, from the at least two putative values, the particular output threshold value for the particular slice comprises: (mental process – evaluation or judgement, a person can manually select, from the two values, the particular threshold value)
determining that at least two candidate values of the at least two putative values, when applied to the plurality of model outputs corresponding to the particular slice, result in the same maximal value of the at least one metric for the particular slice (mental process – evaluation or judgement, a person can manually determine that two values result in the same maximal value of the metric)
selecting, from the at least two candidate values, the particular output threshold value for the particular slice by determining which of the at least two candidate values, when applied to the plurality of model outputs corresponding to the particular slice, result in a maximal value of the at least one secondary metric for the particular slice (mental process – evaluation or judgement, a person can manually select, from the two values, the particular threshold value that results in a maximal value)
2A Prong 2: The judicial exception is not integrated into a practical application.
obtaining at least one secondary metric for the particular slice (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
obtaining at least one secondary metric for the particular slice (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 7:
2A Prong 1: The claim recites a judicial exception.
wherein determining which of the at least two putative values, when applied to the plurality of model outputs corresponding to the particular slice, result in a maximal value of the at least one metric for the particular slice comprises (mental process – evaluation or judgement, a person can manually determine which of the two values results in a maximal value)
applying respective weights to the model outputs corresponding to the particular slice when computing values of the at least one metric for the at least two putative values (mental process – evaluation or judgement, a person can manually apply respective weights to the model outputs corresponding the slice)
With respect to claims 8 and 19:
2A Prong 1: The claim recites a judicial exception.
further comprising: for each slice of the plurality of slices, determining a respective output calibration, and (mental process – evaluation or judgement, a person can manually determine an output calibration for each slice)
applying the plurality of raw model outputs to a particular output calibration determined for the particular slice to generate the plurality of model outputs corresponding to the particular slice (mental process – evaluation or judgement, a person can manually apply raw model outputs to an output calibration)
2A Prong 2: The judicial exception is not integrated into a practical application.
wherein applying each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of model outputs corresponding to the particular slice comprises (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying each input sample to the trained model, to generate model outputs)
applying the each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of raw model outputs (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying the input sample to the trained model, to generate raw model outputs)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
wherein applying each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of model outputs corresponding to the particular slice comprises (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying each input sample to the trained model, to generate model outputs)
applying the each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of raw model outputs (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying the input sample to the trained model, to generate raw model outputs)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 9:
2A Prong 1: The claim recites a judicial exception.
wherein determining the particular output calibration for the particular slice comprises determining a Platt calibration for the plurality of raw model outputs (mental process – evaluation or judgement, a person can manually determine a Platt calibration for raw model outputs)
With respect to claims 10 and 20:
2A Prong 1: The claim recites a judicial exception.
wherein determining the particular output calibration for the particular slice comprises (mental process – evaluation or judgement, a person can manually determine the output calibration)
using each raw model output to update a corresponding bucket of a plurality of buckets, wherein each bucket of the plurality of buckets represents a respective non-overlapping range of possible model output values (mental process – evaluation or judgement, a person can manually use each raw model output to update a corresponding bucket of a plurality of buckets )
determining the particular output calibration for the particular slice based on the plurality of buckets (mental process – evaluation or judgement, a person can manually determine the particular output calibration based on the buckets)
With respect to claim 11:
2A Prong 1: The claim recites a judicial exception.
wherein each bucket of the plurality of buckets represents: (i) a respective count of raw model outputs assigned to the bucket and corresponding to a first class of inputs, (ii) a respective count of raw model outputs assigned to the bucket and corresponding to a second class of inputs, (iii) a respective sum of weights of raw model outputs assigned to the bucket and corresponding to a first class of inputs, and (iv) a respective sum of weights of raw model outputs assigned to the bucket and corresponding to a second class of inputs. (mental process – evaluation or judgement, claim 10 recites “updating a corresponding bucket,” which is an abstract idea. Incorporating more details of the buckets does not change the scope of the claim)
With respect to claim 12:
2A Prong 1: The claim recites a judicial exception.
updating the particular output threshold value for the particular slice based on the additional input data (mental process – evaluation or judgement, a person can manually update the particular threshold value based on the additional input data)
2A Prong 2: The judicial exception is not integrated into a practical application.
obtaining additional input data corresponding to the particular slice (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
obtaining additional input data corresponding to the particular slice (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 13:
2A Prong 1: The claim recites a judicial exception.
determining an updated output threshold value for the particular slice by (mental process – evaluation or judgement, a person can manually determine an updated threshold value)
determining at least two updated putative values of the updated output threshold value that, when applied to the plurality of updated model outputs corresponding to the particular slice, satisfy the at least one constraint for the particular slice (mental process – evaluation or judgement, a person can manually determine two updated putative values satisfying the constraint)
selecting, from the at least two updated putative values, the updated output threshold value for the particular slice by determining which of the at least updated two putative values, when applied to the plurality of updated model outputs corresponding to the particular slice, result in a maximal value of the at least one metric for the particular slice. (mental process – evaluation or judgement, a person can manually select the updated output threshold value resulting in a maximal value of the metric)
2A Prong 2: The judicial exception is not integrated into a practical application.
obtaining an updated machine learning model (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting)
applying each input sample of the input data that corresponds to the particular slice to the updated machine learning model to generate a plurality of updated model outputs corresponding to the particular slice (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying the input sample to the updated model)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
obtaining an updated machine learning model (insignificant extra-solution activity – MPEP 2106.05(g), (3) data gathering and outputting, and WURC: receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 - MPEP 2106.05(d)(II)(i))
applying each input sample of the input data that corresponds to the particular slice to the updated machine learning model to generate a plurality of updated model outputs corresponding to the particular slice (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of applying the input sample to the updated model)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 14:
2A Prong 2: The judicial exception is not integrated into a practical application.
wherein obtaining the trained machine learning model comprises training the machine learning model using at least one input sample of the of the input data that corresponds to each slice of the plurality of slices (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of training the model)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
wherein obtaining the trained machine learning model comprises training the machine learning model using at least one input sample of the of the input data that corresponds to each slice of the plurality of slices (mere instructions to apply an exception – MPEP 2106.05(f), (3) The particularity or generality of the application of the judicial exception; high level recitation of training the model)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gendler ("Adversarially robust conformal prediction" 20220128) and Gueret (US 20220138632 A1, filed on 2020-10-29) teach certain limitations of claim 1, as set forth in the following table.
A computer-implemented method comprising:
obtaining input data, wherein the input data includes a plurality of input samples;
(Gueret, [0023] "As further shown in FIG. 1A, and by reference number 110, the model calibration system may receive calibration data and the rules."; [0014] "the model calibration system may receive calibration data [input data] associated with a plurality of units [a plurality of input samples] and a set of rules")
assigning each input sample of the input data to a respective slice of a plurality of slices;
(Gueret, [0026] "As further shown in FIG. 1A, and by reference number 115, the model calibration system may determine groups of calibration data according to rules and feature attributes... The model calibration system may use any suitable technique (e.g., natural language processing, optical character recognition, text recognition, or the like) to analyze, sort, and/or assign calibration data to corresponding groups [assigning each input sample of the input data to a respective slice of a plurality of slices] based on the attributes and/or features of the calibration data and the defined groups in the set of rule.")
for each slice in the plurality of slices,
(Gendler, p. 13, S2 DATA SETS "We evaluate our methods on three widely used image classification data sets: CIFAR10, CIFAR100..., and ImageNet... we... split the rest 10,000 points into two sets of equal size (calibration [slice] and test) 50 times. We use the ILSVRC2012 version of the ImageNet data set... To construct calibration and test sets, we randomly split the 50,000 images of the ILSVRC2012 validation set into calibration [slice] and test sets of equal size, for 50 times."; see Fig. S7, each calibration set of CIFAR10, CIFAR100, and ImageNet [each calibration set in three data sets, each slice in the plurality of slices])
obtaining a respective at least one constraint and a respective at least one metric;
(Gendler, p. 1, 1 INTRODUCTION "P[Y_n+1 ∈ C(X_n+1] ≥ 1- α (1) [constraint] For example, it is common to set the desired coverage level 1- α to be 90% or 95%. [constraint, user-specified level]"; p. 3, 2 CONFORMAL PREDICTION "we compute a non-conformity score... We refer to the score from (3) as HPS [e.g. metric] as it was shown to construct homogeneous prediction sets. Analogously, we refer to (4) as APS [e.g. metric] since it tends to yield adaptive prediction sets that reflect better the underlying uncertainty across sub-populations..."; see Fig. S7 smoothed APS and HPS are used for each of the datasets)
obtaining a trained machine learning model;
(Gendler, p. 6, Algorithm 1, line 2 "Train a classifier using A on all samples from I_tr, i.e., f^(X) ← A({(Xi, Yi)} i ∈ I_tr)"; p. 13, S2 DATA SETS "In our experiments, we use the entire training set for fitting the classifiers [obtaining a trained machine learning model]... We use the ILSVRC2012 version of the ImageNet data set... We use the entire ILSVRC2012 training set for fitting the predictive models.")
determining a respective output threshold value for each slice in the plurality of slices, wherein determining a particular output threshold value for a particular slice in the plurality of slices comprises:
(Gendler, p. 4, 3.1 ADVERSARIALLY ROBUST CALIBRATION "... S~(X~_n+1, y) ≤ Q_1-α({S~_i} i∈I_cal)+ M_δ (8)... the prediction set defined above is generated by comparing the test score to an inflated threshold Q_1-α + Mδ"; p. 6, Algorithm 1, line 5 "Q_1-α({S^_i} i∈I_cal)+ δ/σ [e.g. δ/σ = 1/2, 1, 2..., a respective output threshold value, a particular output threshold value] "; see Fig. S7, δ/σ can be 1/8, 1/6, 1/4, 1/2, 1, 2 will determine Q_1-α + δ/σ [threshold value] for each calibration set of CIFAR10, CIFAR100, and ImageNet [each calibration set in three data sets, each slice in the plurality of slices)
applying each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of model outputs corresponding to the particular slice;
(Gendler, p. 6, Algorithm 1, "line 3: S^i = ... [model outputs (smoothed scores)] for all i ∈ I_cal. (12) [applying each input sample i of the input data I_cal (i.e. each calibration sample)]"; p. 3, 2 CONFORMAL PREDICTION "a classifier [the trained machine learning model] f^(x) ∈[0, 1] is fit to the proper training set... we compute a non-conformity score Si = S(Xi, Yi) for each calibration point {(Xi, Yi)}i∈I_cal. This score expresses how well the model prediction f^(X) is aligned with the true label Y, where a lower score implies better alignment. For example, the score from Vovk... is given by S(x, y) = 1-f^(x) (3)... Another example is the score proposed by Romano... S(x, y)... (4)"; p. 5, 3.2 THE RANDOMLY SMOOTHED SCORE "where S~ is a 'smoothed' version of the base score S [model outputs]... Let S~ be the randomly smoothed score function (10) and set Mδ =δ/σ."; providing calibration set to a trained classifier to generate smoothed score S~ [model outputs] (S(x, y), f^(x), S~ are all model outputs))
determining at least two putative values of the particular output threshold value that, when applied to the plurality of model outputs corresponding to the particular slice, satisfy the at least one constraint for the particular slice; and
(Gendler, p. 6, Algorithm 1, "line 4: Compute Q_1-α({S^i} i∈I_cal) := the ...th empirical quantile of {S^i} i∈I_cal. "; p. 3 "Q_1-α({S_i} i∈I_cal) ... (6) is the score positioned ceil ((n+1)(1-α)) in the sorted array of calibration scores Si, i∈I_cal. [when applied to the plurality of model outputs corresponding to the particular slice]"; p. 4, 3.1 ADVERSARIALLY ROBUST CALIBRATION "we propose to construct a prediction set robust to a norm-bounded adversarial attack by applying the following decision rule: Cδ(X_n+1)... (8)... the prediction set defined above is generated by comparing the test score to an inflated threshold Q_1-α + Mδ [the threshold value]... the prediction set Cδ(X_n+1) defined in (8) satisfies P[Y_n+1 ∈ Cδ(X_n+1)] ≥ 1- α [satisfy the constraint]"; p. 6, Algorithm 1, line 5 "Q_1-α({S^_i} i∈I_cal) + δ/σ [see Fig. S7, e.g. δ/σ = 1/2, 1, 2... , two putative values]"; Q_1-α + δ/σ [the particular output threshold value], and δ/σ can be 1/8, 1/6, 1/4, 1/2, 1, 2 [determining at least two putative values of the particular output threshold value], where Q_1-α is determined by applying to the score S^ (or f^) [model outputs], for each calibration set of CIFAR10, CIFAR100, and ImageNet [each calibration set in three data sets, each slice in the plurality of slices])
selecting, from the at least two putative values, the particular output threshold value for the particular slice by determining which of the at least two putative values, when applied to the plurality of model outputs corresponding to the particular slice,
(Gendler, p. 15, S5.1 INFLUENCE OF THE VALUE OF σ "Herein, we extend the discussion from Section 5.1 of the main manuscript and study the effect of the choice of the hyper-parameter σ used to compute the smoothed score...Figure S7 shows how the ratio between σ and δ affects the average set-size and the average coverage obtained by our RSCP method... As can be seen, a ratio of σ/δ = 2 works well for all data sets and the two non-conformity scores. [x-axis in Fig. S7, Mδ = δ/σ, selecting, from the at least two putative values, the particular output threshold value for the particular slice by determining which of the at least two putative values]"; p. 8 Choosing σ for smoothing "In practice, we find that setting in σ = 2δ (10) yields informative prediction sets, as illustrated in Figure 3."); see Fig. 3 and Fig. S7 selecting σ/δ = 2 (i.e. Mδ = δ/σ = 1/2) from 1/8, 1/6, 1/4, 1/2, 1, 2, when applied to the sorted scores)
result in a maximal value of the at least one metric for the particular slice; and
providing the respective output threshold value determined for each slice in the plurality of slices for application with the trained machine learning model.
(Gueret, [0070] "the calibrated machine learning model 305 may determine and/or predict a value of 0.35 for the target variable of Target Error [the respective output threshold value determined] for the new observation, as shown by reference number 315… such as qualifying Unit Z, as a member of group 1 [for each slice in the plurality of slices] and group 3"; [0071] "if the machine learning system classifies the new observation in a particular cluster (e.g., a cluster of observations associated with units that are members of a same set of groups), then the machine learning system may provide a recommendation and/or a prediction, analyze the groups [for application with the trained machine learning model] based on a target error associated with a particular set of groups [the determined threshold value for the slice] and/or objectives of the particular cluster.")
PNG
media_image1.png
852
930
media_image1.png
Greyscale
Gendler teaches selecting, from multiple values, a ratio of σ/δ = 2 (i.e. δ/σ = 1/2 in the Figure S7) will result in good marginal coverage (e.g. > 90) and good set size (smaller and more informative sets). However, selecting σ/δ = 2 (i.e. δ/σ = 1/2 in the Figure S7) does not result in a maximal value, because the value σ/δ = 1/2 (i.e. δ/σ = 2 in the Figure S7) - not σ/δ = 2 (i.e. δ/σ = 1/2 in the Figure S7) results in a maximal value, a maximal marginal coverage.
(Gendler's method (or any standard conformal prediction) is aimed at achieving a nominal coverage level (e.g. > 90%), not a maximal coverage (e.g. 100%). Gendler's method is further aimed at choosing the best setting of Mδ with Q_1-α (hyperparameters with conformal threshold) that will achieve both nominal coverage and informative prediction sets.)
Therefore, combining Gendler and Gueret does not teach:
for each slice in the plurality of slices, obtaining a respective at least one constraint and a respective at least one metric;
obtaining a trained machine learning model;
determining a respective output threshold value for each slice in the plurality of slices, wherein determining a particular output threshold value for a particular slice in the plurality of slices comprises:
applying each input sample of the input data that corresponds to the particular slice to the trained machine learning model to generate a plurality of model outputs corresponding to the particular slice;
determining at least two putative values of the particular output threshold value that, when applied to the plurality of model outputs corresponding to the particular slice, satisfy the at least one constraint for the particular slice; and
selecting, from the at least two putative values, the particular output threshold value for the particular slice by determining which of the at least two putative values, when applied to the plurality of model outputs corresponding to the particular slice, result in a maximal value of the at least one metric for the particular slice;
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SU-TING CHUANG whose telephone number is (408)918-7519. The examiner can normally be reached Monday - Thursday 8-5 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SU-TING CHUANG/Examiner, Art Unit 2146