DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/14/2026 has been entered.
Status of the Claims
Claims 1-2, 13-14, 16, 18-19 have been amended. Claims 1-20 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The claims at a high level recite estimating metrics for AI model.
Step 1: Does the Claim Fall within a Statutory Category?
Yes. Claims 1-20 recite a method and a system and therefore, are directed to the statutory class of machine and a product.
The USPTO Guidance recites:
(1) any judicial exceptions, including certain groupings of abstract ideas (i.e., mathematical concepts, certain methods of organizing human activity such as a fundamental economic practice, or mental processes) (Step 2A, Prong 1); and
(2) additional elements that integrate the judicial exception into a practical application (Step 2A, Prong 2). MPEP §§ 2106.04(a), (d).
Only if the claim (1) recites a judicial exception and (2) does not integrate that exception into a practical application, do we then look in Step 2B to whether the claim:
(3) adds a specific limitation beyond the judicial exception that is not “well-understood, routine, conventional” in the field; or
(4) simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception. MPEP § 2106.05(d).
Step 2A, Prong One: Is a Judicial Exception Recited?
First, determine whether the claims recite any judicial exceptions, including certain groupings of abstract ideas (i.e., mathematical concepts, certain methods of organizing human activity, or mental processes). MPEP § 2106.04(a).
Claim 1 recites –
▪ accessing, at processing circuitry of one or more computing machines, an artificial intelligence (AI) model (Amount to “Apply it”. Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f). Examiner’s note: high level application of using artificial intelligence for data processing is merely invoking a computer component to apply the exception),
▪ in-sample (IS) dataset that is labeled, and an out-of-sample (OOS) dataset that is unlabeled, the IS dataset storing datapoints, each datapoint comprising an IS input value and corresponding IS output value, the OOS dataset storing OOS datappoints having OOS input values but not corresponding OOS output value (Abstract Idea of a mental process, see MPEP § 2106.04(a)(2)(III). Under the broadest reasonable interpretation, this limitation is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion) — a user can sample and label data);
▪ calculating, via importance sampling weights of multiple datapoints in the IS dataset to generate a weighted IS dataset, the weight of each from the multiple datapoints datapoint being a numerical value based on a likelihood that the datapoint is associated with the IS dataset and a likelihood that the datapoint is associated with the OOS dataset (Abstract Idea of a mental process, such as mathematical evaluations, which performs the determination, thereby further defining the abstract idea. A human being may use this mathematical calculation to facilitate the mental evaluation in order to arrive at the necessary determination. This claim limitation appears to recite both a mathematical formula and mental process. See MPEP § 2106.04(a)(2)(III). Under the broadest reasonable interpretation, this limitation is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion. Specifically the limitations recite mathematical evaluations, which performs the determination, thereby further defining the abstract idea. A human being may use this mathematical calculation to facilitate the mental evaluation in order to arrive at the necessary determination. This claim limitation appears to recite both a mathematical formula and mental process);
▪ calculating an estimated performance metric of the Al model on the OOS dataset based on the weights of at least a subset of datapoints in the weighted IS dataset (i.e. additional generic mathematical calculations and do not represent significantly more than the abstract idea. See at least MPEP § 2106.05(a) ("Improvements to the Functioning of a Computer or to Any Other Technology or Technical Field")); and
▪ providing, using the processing circuitry, an output representing the estimated performance metric of the Al model on the OOS dataset (An abstract idea of a mental process, see MPEP § 2106.04(a)(2)(III). Under the broadest reasonable interpretation, this limitation is an abstract idea of “a mental process”— a user can determine performance metrics and generate an output);
▪ calibrating the Al model using isotonic regression based on the weights of the multiple datapoints, thereby enabling estimation of a goodness of the Al model (is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion. Here, an additional mathematical /logical reasoning). (i.e. additional generic mathematical calculations and observations and do not represent significantly more than the abstract idea (i.e. additional generic mathematical calculations and do not represent significantly more than the abstract idea. See at least MPEP § 2106.05(a) ("Improvements to the Functioning of a Computer or to Any Other Technology or Technical Field"));
▪ generating predicted output values for the OOS dataset based on the calibrated Al model (is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion. Here, an additional logical reasoning and evaluation of data. Generic computer implementations, such as data output, does not provide significantly more than the abstract idea).
These limitations, based on their broadest reasonable interpretation, recite a mental process, i.e. a judicial exception. For these reasons, the independent claim 1, as well as independents claims 13 and 18, which include limitations commensurate in scope with claim 1, recite a judicial exception.
A method, like the claimed method, “a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.” See Digitech Image Techs, LLC v. Elecs. for Imaging, Inc., 758 F.3d 1344, 1351 (Fed. Cir. 2014). See Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350 (Fed. Cir. 2016) where collecting information, analyzing it, and displaying results from certain results of the collection and analysis was held to be an abstract idea. See In re Meyer, 688 F.2d 789, 795—96 (CCPA 1982), which held that “a mental process that a neurologist should follow” when testing a patient for nervous system malfunctions was not patentable.
Accordingly, the claims recite an abstract idea.
Step 2A, Prong Two: Is the Abstract Idea Integrated into a Practical Application?
Next determine whether the claims recite additional elements that integrate the judicial exception into a practical application (see MPEP §§ 2106.05(a)-(c), (e)-(h)). To integrate the exception into a practical application, the additional claim elements must, for example, improve the functioning of a computer or any other technology or technical field (see MPEP § 2106.05(a)), apply the judicial exception with a particular machine (see MPEP § 2106.05(b)), or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment (see MPEP § 2106.05(e)).
Additional elements:
▪ processing circuitry of one or more computing machines (Amount to “Apply it”. Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f). Examiner’s note: high level application of using machine learning model to process data amount to merely invoking a computer component to apply the exception);
▪ an artificial intelligence (AI) model (Amount to “Apply it”. Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f). Examiner’s note: high level application of using machine learning model to process input data and generate an output).
▪ providing, using the processing circuitry, an output (Adding insignificant extra-solution activity to the judicial exception - see MPEP § 2106.05(g));
▪ calibrating the Al model (Amounts to apply mathematical evaluations to update predictions. Examiner’s note: high level application of updating a machine learning model with additional calculations amount to merely invoking a computer component to apply the exception.)
▪ enabling estimation of a goodness of the Al model (Intended use of the disclosed calibrations.)
The term “additional elements” for claim features, limitations, or steps that the claim recites beyond the identified judicial exception. Claim 13 additionally recite “a memory comprising instructions; and one or more computer processors” and claim 18 recites “tangible machine-readable storage medium.” However, claims do not recite any improvements to these additional elements, nor does the claims recite any particularly programmed or configured computer system, device, or machine learning. Rather, the additional elements in claims 1, 13 and 18 serve merely to automate the abstract idea. See Int’l Bus. Machs. Corp. v. Zillow Group, Inc., 50 F. 4" 1371, 1382 (Fed. Cir. 2022) (“[A] patent that ‘automate[s] “pen and paper methodologies” to conserve human resources and minimize errors’ is a ‘quintessential “do it on a computer” patent’ directed to an abstract idea.”) (quoting Univ. of Fla. Rsch. Found., Inc. v. Gen. Elec. Co., 916 F.3d 1363, 1367 (Fed. Cir. 2019)). Therefore, none of these recited additional elements, whether considered individually or in combination, integrates the judicial exception into a practical application.
The additional elements listed above that relate to computing components are recited at a high level of generality (i.e., as generic components performing generic computer functions such as communicating and processing known data) such that they amount to no more than mere instructions to apply the exception using generic computing components. Simply implementing the abstract idea on a generic computer is not a practical application of the abstract idea. Additionally, the claims do not purport to improve the functioning of the computer itself. There is no technological problem that the claimed invention solves. Rather, the computer system is invoked merely as a tool. Accordingly, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Therefore, these claims are directed to an abstract idea.
Step 2B: Does the Claim Provide an Inventive Concept?
Next, determine whether the claims recite an “inventive concept” that “must be significantly more than the abstract idea itself, and cannot simply be an instruction to implement or apply the abstract idea on a computer.” BASCOM Glob. Internet Servs., Inc. v. AT&T Mobility LLC, 827 F.3d 1341, 1349 (Fed. Cir. 2016); see MPEP § 2106.05(d). There must be more than “computer functions [that] are “well-understood, routine, conventional activit[ies]’ previously known to the industry.” Alice Corp. v. CLS Bank Int'l, 573 U.S. 208, 225 (2014) (second alteration in original) (quoting Mayo Collaborative Servs. v. Prometheus Labs., Inc., 566 U.S. 66, 73 (2012)); see MPEP § 2106.05(d).
Step 2B: The additional elements are not sufficient to amount to significantly more than the judicial exception. (see MPEP 2106.05(d)(Il). Taking the claim elements separately, the function performed by the computer at each step of the process is purely conventional. Using a computer and associated computer network to obtain data, use data to identify other data, and comparing data, are some of the most basic functions of a computer. All of these computer functions are well-understood, routine, conventional activities previously known to the industry. The method claims do not, for example, purport to improve the functioning of the computer itself. Nor do they effect an improvement in any other technology or technical field. Instead, the claims at issue amount to nothing significantly more than an instruction to apply the abstract idea of displaying, processing and storing data using some unspecified, generic computer).
Note, that in similar case, such as Collecting information, analyzing it, and displaying certain results of the collection and analysis (Electric Power Group), the Courts have identified that the additional elements of displaying and analyzing data, as shown in the independent claims 1, 13 and 18 do not amount to significantly more than the judicial exception. Consequently, that is not enough to transform an abstract idea into a patent-eligible invention.
No “inventive concept” sufficient to transform the abstract method of organizing human activity into a patent-eligible application. See MPEP § 2106.05. Rather, the additional elements identified above are merely well-understood, conventional computer components, as confirmed by the Specification. See MPEP § 2106.05(d)(1). For example, the Specification refers to the additional elements in generic terms.
As discussed above with respect to integration of the abstract idea into a practical application, the additional elements relating to computing components amount to no more than applying the exception using a generic computing components. Mere instructions to apply an exception using a generic computing component cannot provide an inventive concept. Furthermore, the broadest reasonable interpretation of the claimed computer components (i.e., additional elements) includes any generic computing components that are capable of being programmed to communicate and process known data.
Additionally, the computer components are used for performing insignificant extra-solution activity and well understood, routine, and conventional functions. For example, the claimed processor and machine learning merely communicates and processes known data. Activities such as these are insignificant extra-solution activity and, therefore, well understood, routine, and conventional. See MPEP 2106.05(d); see also, e.g., OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d at 1363, 115 USPQ2d at 1092-93 (Presenting offers to potential customers and gathering statistics generated based on the testing about how potential customers responded to the offers; the statistics are then used to calculate an optimized price); CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011) Ultramercial, Inc. v. Hulu, LLC, 772 F.3d at 715, 112 USPQ2d at 1754 (Consulting and updating an activity log); Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354-55, 119 USPQ2d 1739, 1742 (Fed. Cir. 2016) (Selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display); Apple, Inc. v. Ameranth, Inc., 842 F.3d 1229, 1244, 120 USPQ2d 1844, 1856 (Fed. Cir. 2016) (Recording a customer’s order); Return Mail, Inc. v. U.S. Postal Service, -- F.3d --, -- USPQ2d --, slip op. at 32 (Fed. Cir. August 28, 2017) (Identifying undeliverable mail items, decoding data on those mail items, and creating output data); Furthermore, limitations such as integrating account details are well-understood, routine, and conventional activity. See Alice Corp., 134 S. Ct. at 2359, 110 USPQ2d at 1984 (creating and maintaining "shadow accounts"); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log).
Independent system claim 1, 13 and 18 contain the identified abstract ideas, with the additional elements of a processor, hardware and the media, which is a generic computer component, and thus not significantly more for the same reasons and rationale above. Accordingly, independent claims 1, 10 and 19 are patent ineligible because they are directed to an abstract idea that does not recite an inventive concept that amounts to significantly more than the abstract idea.
Dependent 2-12, 14-17, 19-20 claims further describe the abstract idea. The additional elements of the dependent claims fail to integrate the abstract idea into a practical application and do not amount to significantly more than the abstract idea. Thus, as the dependent claims remain directed to a judicial exception, and as the additional elements of the claims do not amount to significantly more, the dependent claims are not patent eligible.
With respect to claims 2, 14 and 19:
Step 2A Prong 1: the claims recite a judicial exception (an abstract idea)
▪ calculating, for the datapoint x, a probability pis(x) that the datapoint x is associated with the IS dataset using density estimation; calculating, for the datapoint x, a probability poos(x) that the datapoint x is associated with the OOS dataset using density estimation; and calculating the weight of the datapoint x as the poos(x) divided by the pis(x) (is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion. Here, an additional mathematical /logical reasoning). (i.e. additional generic mathematical calculations and do not represent significantly more than the abstract idea. See at least MPEP § 2106.05(a) ("Improvements to the Functioning of a Computer or to Any Other Technology or Technical Field")).
Step 2A Prong 2: the additional elements that are not sufficient to integrate the judicial exception into a practical application. Step 2B: the additional element is not sufficient to amount to significantly more than the judicial exception.
With respect to claims 3, 15 and 10:
Step 2A Prong 1: the claims recite a judicial exception (an abstract idea)
▪ calculating, for a given model input value, a probability that the given model input value is associated with the IS dataset (pis(x)) using density estimation; calculating, for the given model input value, a probability that the given model input value is associated with the OOS dataset (poos(x)) using density estimation; and calculating a probability that the given model input value corresponds to a given output value (y) for the OOS dataset (poos(x,y)) based on the probability that the given model input value is associated with the OOS dataset divided by the probability that the given model input value is associated with the IS dataset (poos(x) / pis(x)), wherein the estimated performance metric of the Al model on the OOS dataset is calculated based on the probability that the given model input value corresponds to the given output value (Abstract Idea of a mental process, such as a mathematical evaluation, which performs the determination, thereby further defining the abstract idea. A human being may use this mathematical calculation to facilitate the mental evaluation in order to arrive at the necessary determination. This claim limitation appears to recite both a mathematical formula and mental process. Under the broadest reasonable interpretation, the obtaining/determining probability distribution and divergence, as drafted, is an abstract idea of “a mental process” because it recites a process that can be performed in the human mind (i.e., observation, determination, evaluation, judgment, and opinion) — a user can logically perform such mathematical calculations.)
Step 2A Prong 2: the additional elements that are not sufficient to integrate the judicial exception into a practical application.
Additional elements:
model input / values, Al model (mathematical evaluation, which performs the determination, by applying AI model, thereby further defining the abstract idea. Amount to mere instruction to apply the abstract idea using a generic computer component. A mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).)
Step 2B: the additional element is not sufficient to amount to significantly more than the judicial exception.
With respect to claims 4-7, 9-11, and 16-17:
Step 2A Prong 1: the claims recite a judicial exception (an abstract idea)
▪ Claims recite further additional mathematical calculations, such as density estimation of the IS dataset and the OOS dataset, computing a probability, threshold amount of concept drift, a quotient, logistic regression, a quantitative input influence, precision, recall, F1-score, receiver operating characteristic area under the curve (ROC-AUC), and classification accuracy, a quantity defined by a ground truth label and a predicted label probability (Mathematical evaluations, which performs the determination, thereby further defining the abstract idea. A human being may use this mathematical calculation to facilitate the mental evaluation in order to arrive at the necessary determination. This claim limitation appears to recite both a mathematical formula and mental process);
Step 2A Prong 2: the additional elements that are not sufficient to integrate the judicial exception into a practical application. Additional elements: no additional elements recited. Step 2B: the additional element is not sufficient to amount to significantly more than the judicial exception.
With respect to claims 8 and 12:
Step 2A Prong 1: the claims recite a judicial exception (an abstract idea)
▪ a generative adversarial network (GAN) that distinguishes between datapoints in the IS dataset and datapoints in the OOS dataset (Amount to “Apply it”. Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f). Examiner’s note: high level application of using generative adversarial network to process data amount to merely invoking a computer component to apply the exception);
▪ a multithreaded processing unit, and the weights of multiple datapoints in the labeled IS dataset are modified in parallel using multiple threads of the multithreaded processing unit (Amount to “Apply it”. Merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f). Examiner’s note: such multithreaded processing is a generic computer function of processing data in parallel that are well-understood, routine, and conventional activities previously known to the industry.)
Step 2A Prong 2: the additional elements that are not sufficient to integrate the judicial exception into a practical application. Step 2B: the additional element is not sufficient to amount to significantly more than the judicial exception.
Dependent claims 2-12, 14-17, 19-20 do not recite additional limitations that demonstrate integration of the abstract idea into a practical application or an inventive concept that amounts to significantly more than the abstract idea. As such, the claims are not patent eligible.
Claim Objections
Claims 1, 8 and 13 are objected to because of the following informalities:
Intendent claims recite “OOS datappoints”, which should be corrected to – “OOS datapoints.”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 10, 12-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over in view of Zhong et al. (US 20210073665) in view of Al-Turki et al. (US 10,678,196) and in further view of Pampari et al. “Unsupervised Calibration under Covariate Shift.”
Regarding claim 1, Zhong teaches a method comprising:
accessing, at processing circuitry of one or more computing machines, an artificial intelligence (Al) model, in-sample (IS) dataset that is labeled ([0051] “dx represents a domain label of the sample x, that is, if x is from the source dataset “, [0031], [0038]), and an out-of-sample (OOS) dataset that is unlabeled, the IS dataset storing datapoints ([0042] “samples in the target dataset do not have labels”, [0082] “a target dataset is unlabeled (i.e., in a case where a label of the target dataset is unknown”),
each datapoint comprising an IS input value and corresponding IS output value, the OOS dataset storing OOS datappoints having OOS input values
calculating, via importance sampling weights (see NOTE) of multiple datapoints in the IS dataset to generate a weighted IS dataset ([0041]-[0042] “weighted accuracy score of the model with respect to the sample in the source dataset is calculated” and “estimated accuracy of the model with respect to the target dataset, according to the weighted accuracy score” - i.e. weight of each datapoint creates weighted dataset), the weight of each datapoint from the multiple datapoints being a numerical value based on a likelihood that the datapoint is associated with the IS dataset and a likelihood that the datapoint is associated with the OOS dataset ([0030]-[0031], [0039], [0046], [0072], [0074]);
calculating an estimated performance metric of the Al model on the OOS dataset based on the weights of at least a subset of datapoints in the weighted IS dataset ([0030], [0038]-[0039], [0041], [0047], [0072]-[0073], F2);
calibrating the Al model
Zhong does not explicitly teach, however Al-Turki discloses the OOS dataset storing OOS datappoints having OOS input values but not corresponding OOS output values (C5L32-35 “a data sample with both input and output information is denoted as “labeled” data, whereas a data sample that only incorporates input variables is denoted as “unlabeled” data”, C1 “unlabeled data samples has only input information”).
Al-Turki further discloses generating predicted output values for the OOS dataset based on the calibrated Al model (C3L46-60, C7L4-5, 12-20 “may include a prescriptive action comprising a change to a set point … or a change to some other value, parameter, configuration … automatically … cause the change to be implemented … automatically optimizing the performance and output”, C12L37-64).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhong to include OOS datappoints having OOS input values but not corresponding OOS output value as disclosed by Al-Turki. Doing so would provide reliable and accurate predictions, especially when the number of labeled data samples is small (Al-Turki C2L44-46).
Zhong does not explicitly teach, however Pampari discloses calibrating the Al model using isotonic regression based on the weights of the multiple datapoints (p.2 C1L6-7, p.5 C2 “incorporate the weighted calibration loss for isotonic regression to optimize the calibrator on the source data”.
NOTE Pampari further explicitly discloses calculating, via importance sampling weights (see NOTE) of multiple datapoints in the IS dataset to generate a weighted IS dataset (see pages 5-6 IMPORTANCE SAMPLING FOR CALIBRATION UNDER COVARIATE SHIFT).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhong to include isotonic regression based on the weights of the multiple datapoints as disclosed by Pampari. Doing so would improve the generalization or predictive accuracy on the unseen target data (Pampari p.5 C1L1-2).
Further, note – previously cited Sampaio et al. (US 20210374614) and Li et al. "Learning to calibrate and rerank multi-label predictions," likewise disclose calibrating the Al model using isotonic regression based on the weights of the multiple datapoints (Li p.224, p.227, p.229 “the top set candidates with their scores and additional features … and select the one with the highest calibrated confidence as the final prediction”, Table 3, Sampaio [0084]-[0085], [0087]-[0088], [0091]-[0098]) and can be interchanged with the Pampari’s reference and further obviate the teachings of Zhong.
Claims 13 and 18 recite substantially the same limitations as claim 1, and is rejected for substantially the same reasons.
Regarding claims 2, 14 and 19, Zhong as modified teaches the method, the system and the medium, wherein calculating the weight of a datapoint x via the importance sampling comprises:
calculating, for the datapoint x, a probability pis(x) that the datapoint x is associated with the IS dataset using density estimation (Al-Turki C7L51-60, C8L42-67, C9L45-55, C11L11-45, Zhong [0039], [0039], [0043], [0046]-[0047], [0051]);
calculating, for the datapoint x, a probability poos(x) that the datapoint x is associated with the OOS dataset using density estimation (Al-Turki C12L50-65, Zhong [0039], [0043]); and
calculating the weight of the datapoint x as the poos(x) divided by the pis(x)( Al-Turki C9L1-65 – C10L1-67, Zhong [0030], [0041], [0046]-[0047]).
NOTE While the prior art might not exactly disclose using the formula claimed, the particular elements are obvious, and any particular equation would be an obvious to try combination of elements in order to achieve a predictable results. The use of a particular formula would be based on the particular design choices based on the goals and tradeoffs for those formulas, as is known in the art. See MPEP 2143.
Regarding claims 3, 15 and 10, Zhong as modified teaches the method, the system and the medium, wherein the importance sampling comprises density estimation of the IS dataset and the OOS dataset (Zhong [0030], [0039], [0043], [0046]-[0047], Al-Turki C7L51-60, C8L42-67, C9L45-55, C11L11-45).
Regarding claims 4, and 16, Zhong as modified teaches the method and the system, wherein the importance sampling comprises training a discriminator engine to discriminate between datapoints in the IS dataset and OOS datapoints in the OOS dataset by computing a probability that a given datapoint belongs in the IS dataset rather than the OOS dataset (Al-Turki C12L55-65, Zhong [0039], [0045]-[0046]).
Regarding claims 5, and 17, Zhong as modified teaches the method and the system, wherein the OOS dataset has at least a first threshold amount of data drift from the IS dataset and at most a second threshold amount of concept drift from the IS dataset (Zhong [0047]-[0048], [0055], [0068]).
Regarding claim 6, Zhong as modified teaches the method of claim 5, wherein: the discriminator engine computes a quotient between a probability that a given datapoint is in the OOS dataset and a probability that the given datapoint is in the IS dataset, the probability that the given datapoint is in the OOS dataset is computed using density estimation, and the probability that the given datapoint is in the IS dataset is computed using density estimation (Zhong [0031], [0039], [0041]-[0042], [0045]-[0046], [0051]-[0052] “if x is from the source dataset… then d=1 , otherwise if x is from the target dataset (x-Dr), then dx =0”, [0072], Al-Turki C7L51-60, C8L42-67, C9L45-55, C11L11-45).
Regarding claim 7, Zhong as modified teaches the method of claim 5, wherein the discriminator engine leverages a logistic regression model that distinguishes between datapoints in the IS dataset and datapoints in the OOS dataset (Al-Turki C9L42-47, C19L37-40, Zhong [0031], [0051]).
Regarding claim 10, Zhong as modified teaches the, wherein the performance metric comprises one or more of: precision, recall, F1-score, receiver operating characteristic area under the curve (ROC-AUC), and classification accuracy (Zhong [0013], [0030], [0036], [0099]).
Regarding claim 12, Zhong as modified teaches the method of claim 1, wherein: the processing circuitry comprises a multithreaded processing unit, and the weights of multiple datapoints in the labeled IS dataset are modified in parallel using multiple threads of the multithreaded processing unit (Zhong [0086]).
Claims 8, 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhong as modified and further in view of SINGH et al. (US 20220188568).
Regarding claim 8, Zhong as modified does not explicitly teach, however SINGH discloses wherein the discriminator engine leverages a generative adversarial network (GAN) that distinguishes between datapoints in the IS dataset and datapoints in the OOS dataset ([0041], [0097]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Zhong as modified to include GAN as disclosed by SINGH. Doing so provides an effective method for computer-based mining of minority-class data samples (SINGH [0004]).
Regarding claim 11, Zhong as modified does not explicitly teach, however SINGH discloses, wherein the performance metric comprises a quantity defined by a ground truth label and a predicted label probability (SINGH [0033], [0036], [0054], [0078]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Zhong as modified to include ground truth as disclosed by SINGH. Doing so provides an effective method for computer-based mining of minority-class data samples (SINGH [0004]).
Claim 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhong as modified and further in view of Datta et al. (US 20180121817).
Regarding claim 9, Zhong as modified does not explicitly teach, however Datta discloses wherein the discriminator engine computes, for one or more features of the IS dataset and the OOS dataset, a quantitative input influence (QII) score for predicting whether a feature value for the one or more features are likely to be associated with the IS dataset or the OOS dataset ([0040], [0052], [0063], [0213]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Zhong as modified to include quantitative input influence as disclosed by Datta. Doing so would help to assess the decision making by machine learning systems (Datta [0003]).
Claims 5 and 17 is/are additionally or alternatively rejected under 35 U.S.C. 103 as being unpatentable over Zhong as modified and further in view of KAMULETE et al. (US 20200410403).
Regarding claims 5 and 17, if Zhong as modified does not explicitly teach, however KAMULETE discloses teaches the method and the system, wherein the OOS dataset has at least a first threshold amount of data drift from the IS dataset and at most a second threshold amount of concept drift from the IS dataset ([0069], [0121]-[0122], [0159]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Zhong as modified to include threshold amount of concept drift as disclosed by KAMULETE. Doing so provide additional evaluations of data sets.
◊ Claims 1-2, 13-14 and 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over in view of Marcheret (US 20130254153) in view of Masashi Sugiyama “Covariate Shift Adaptation by Importance Weighted Cross Validation” and in further view of Sampaio et al. (US 20210374614).
Regarding claim 1, Marcheret teaches a method comprising:
accessing, at processing circuitry of one or more computing machines, an artificial intelligence (Al) model, in-sample (IS) dataset that is labeled, and an out-of-sample (OOS) dataset that is unlabeled, the IS dataset storing datapoints ([0039]),
each datapoint comprising an IS input value and corresponding IS output value, the OOS dataset storing OOS datappoints having OOS input values but not corresponding OOS output values ([0034-[0035]);
calculating
calculating an estimated performance metric of the Al model on the OOS dataset based on the weights ([0124], [0126]) of at least a subset of datapoints in the weighted IS dataset ([0086], [0090], [0107]-[0108]);
providing, using the processing circuitry, an output representing the estimated performance metric of the Al model on the OOS dataset ([0084], [0126]);
calibrating the Al model using logistic regression ([0078], [0123]) based on the weights of the multiple datapoints, thereby enabling estimation of a goodness of the Al model ([0075], [0084], [0087]); and
generating predicted output values for the OOS dataset based on the calibrated Al model ([0079], [0087]).
Marcheret does not explicitly teach, however Sugiyama discloses calculating, via importance sampling weights of multiple datapoints in the IS dataset to generate a weighted IS dataset (p.2 C1), the weight of each datapoint from the multiple datapoints being a numerical value based on a likelihood that the datapoint is associated with the IS dataset and a likelihood that the datapoint is associated with the OOS dataset (p.987-988 ¶2.2, p.983 ¶3).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Marcheret to importance sampling weights of multiple datapoints in the IS dataset to generate a weighted IS dataset as disclosed by Sugiyama. Doing so would allow for a ML model can be trained and quickly improved while making an efficient use of labeling resources (Sugiyama [0030]).
Marcheret does not explicitly teach, however Sampaio discloses calibrating the Al model ([0083]) using isotonic regression based on the weights of the multiple datapoints, thereby enabling estimation of a goodness of the Al model ([0084]-[0085]) and generating predicted output values for the OOS dataset based on the calibrated Al model ([0087]-[0088], [0091]-[0098]).
It would have been obvious to one of ordinary skill in the art at the time of invention to modify the teachings of Marcheret to include isotonic regression for calibrating AI as disclosed by Sampaio. Doing so would allow for a ML model can be trained and quickly improved while making an efficient use of labeling resources (Sampaio [0030]).
Regarding claims 2, 14 and 19, Marcheret as modified teaches the method, the system and the medium, wherein calculating the weight of a datapoint x via the importance sampling comprises:
calculating, for the datapoint x, a probability pis(x) that the datapoint x is associated with the IS dataset using density estimation (Marcheret [0052], [0058], [0060], [0063]); calculating, for the datapoint x, a probability poos(x) that the datapoint x is associated with the OOS dataset using density estimation (Marcheret [0057], [0063]-[0066]); and
calculating the weight of the datapoint x as the poos(x) divided by the pis(x)(Marcheret [0081]-[0083]), wherein the estimated performance metric of the Al model on the OOS dataset is calculated based on the weights of the multiple datapoints (Marcheret [0058], [0075]).
Response to Arguments
Applicant's arguments, filed 01/14/2026, with respect to the rejection under 35 USC 101 have been fully considered but they are not persuasive.
The applicant argues –
“the present claims recite a specific technological solution to the well-recognized technical problem of estimating AI model performance on unlabeled out-of-sample data experiencing data drift. The claims do not simply recite mathematical formulas or calculations in the abstract … but rather recite a technological solution to a technological problem-namely, how to estimate and maintain AI model performance when deploying models on unlabeled data that differs statistically from training data.”
The arguments are not persuasive. Estimating a performance of the AI model – is indeed a mathematical concept. The rejection has provided an explanation of which specific mental process each claim limitation falls into. A calibration of the AI model is fundamentally a mental (cognitive-like) process of aligning confidence with actual correctness. Applicant has provided no evidence to support applicant's statements that a human cannot perform the claimed activities as a practical matter.
The additional elements, such as processing circuitry, an artificial intelligence (Al) model and calibrating the model based on weights is a mathematical/ logical reasoning that do not: (1) improve the functioning of a computer or other technology; (2) are not applied with any particular machines (except for a generic computer); (3) do not effect a transformation of a particular article to a different state; and (4) are not applied on any meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception. See MPEP §§ 2 106.05(a) (c), (e) (b). In other words, the aforementioned additional element (or combination of elements) recited in the claims do not integrate the judicial exception into a practical application. See Revised Guidance, 84 Fed. Reg. at 54- 55 ("Prong 2"). Second, the use of computer hardware and/or software components to optimize the processing of data may improve the abstract idea, but, in this context, is not a technological improvement. Appeal Br.5; see Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363. 1367 (Fed. Cir. 2015).
Based on the above rational the applied rejection is maintained.
With regard to the 101 Step 2A, Prong Two evaluation, applicant argues the identification of the recited devices a generic purpose computing devices concerns Step 2B evaluations and not Step 2A, Prong 2. In response to the preceding argument, MPEP 2106.04(d)(I) clearly states that Most of the considerations overlap between step 2A Prong Two and Step 2B. The only consideration explicitly excluded in the Step 2A consideration is the evaluation of well- understood, routine, conventional activities. Please note that the determination of if a device constitutes a general purpose device (MPEP2106.05(b)) is distinct from the evaluation of if a claim limitation is a well-understood, routine, or conventional activity (MPEP2106.05(d)). During the 2A Prong Two evaluation, the office applied the guidance given in MPEP 2106.05(b) to determine that the claimed devices where general purpose devices merely implementing the abstract idea with inn a computing environment.
Based on the above rational the applied rejection is maintained.
The applicant further argues – “the claims cannot be performed entirely in the human mind,” “These operations cannot practically be performed in the human mind, particularly when dealing with real-world AI datasets containing thousands or millions of data points. See Ancora Techs., Inc. V. HTC Am., Inc., 908 F.3d 1343, 1346-47 (Fed. Cir. 2018).”
This argument is not persuasive for the same reasons discussed above. That is, Applicant fails to demonstrate why "calculating, via importance sampling weights of multiple datapoints in the IS dataset to generate a weighted IS dataset, the weight of each datapoint from the multiple datapoints being a numerical value based on a likelihood that the datapoint is associated with the IS dataset and a likelihood that the datapoint is associated with the OOS dataset" cannot, as a practical matter, be performed entirely in the human mind, particularly when processing a dataset storing at least two datapoints.
With respect to Step 2A Prong 2 (Practical Application) Having determined that claim 1 recites at least one abstract idea, evaluation is conducted by first "[i]dentifying whether there are any additional elements recited in the claim beyond the judicial exception(s)," and then "evaluating those additional elements individually and in combination to determine whether they integrate the exception into a practical application." Guidance, 84 Fed. Reg. at 54-55.
Training and retraining (calibrating and recalibrating) computer models are one of the basic practices of the machine learning and merely standard aspects of how machine learning operates. Wherein modern calibration methods are routinely preformed since the 1990s and early 2000s. Particularly a 2025 decision from the Federal Circuit, have determined that simply applying generic machine learning techniques—such as standard model training, updating, or calibration—to new data environments is not patent-eligible under 35 U.S.C. § 101 (Recentive Analytics, Inc. v. Fox Corp. et al., Case No. 23-2437 (Fed. Cir. Apr. 18, 2025) (Dyk, Prost, Goldberg, JJ.). The Federal Circuit concluded that the invention lacked a specific technical improvement and therefore recited an abstract idea. The court noted that iterative training and dynamic adjustments (key aspects of calibration) are inherent to machine learning and not, by themselves, an inventive concept.
Based on recent rulings and USPTO guidance: Not Eligible: Using standard calibration techniques (e.g., Platt scaling, Isotonic regression) to make a conventional model more accurate, or simply applying an existing algorithm to new data. Potentially Eligible: A novel, improved calibration algorithm that specifically reduces memory storage, speeds up inference time, or addresses catastrophic forgetting in a new way.
The present claims do not disclose any novel method for enhancing machine learning algorithms – just their routine application training and retraining (calibrating and recalibrating) by using standard technique, such as Isotonic regression. The claim(s) do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination it does not amount to more than the abstract idea. In particular, the claim only recites one additional element – using a processor to perform both the ranking and determining steps. The processor in both steps is recited at a high-level of generality (i.e., as a generic processing circuitry performing a generic computer functions of accessing data and providing output) such that it amounts no more than mere instructions to apply the exception using a generic computer component.
Based on the above rational the applied rejection is maintained.
With respect to the rejection under 35 USC 103 and Zhong reference, the applicant argues – “The cited paragraphs from Zhong describe a different approach to weight calculation,” “probability density ratio represents a single value characterizing the relationship between the source and target datasets, not two separate likelihoods as required by the claim … probability density ratio r(x) quantifies how much more likely a sample is under the target distribution compared to the source distribution, but this is fundamentally different from separately calculating a likelihood of association with the source dataset and a likelihood of association with the target dataset.”
The arguments are not persuasive. The “density ration represent a single values … not two separate likelihoods” is mathematically false. The probability density ratio is a well-known formula of –
PNG
media_image1.png
25
58
media_image1.png
Greyscale
This explicitly requires two likelihood functions – likelihood under source p(x) and likelihood under target q(x). The probability density ratio cannot exist unless both likelihood exist. Even if only the ration is stored, it is a function from both likelihoods. Thus, Zhong fully teaches two separate likelihoods as required by the claim.
Further in [0046]-[0047] Zhong teaches discriminator which distinguishes between source and target samples and shows calculations of two likelihoods for the same datapoint (i.e. discriminator output provides likelihood of association with target (unlabeled) dataset and likelihood of association with source (labeled) dataset).
Zhong, further explicitly teaches assigning weights to individual datapoints in [0041]-[0042] “weighted accuracy score of the model with respect to the sample in the source dataset is calculated” and “estimated accuracy of the model with respect to the target dataset, according to the weighted accuracy score”. Clearly, each datapoint has associated weight by using density ration, which is what defines being a weighted dataset.
The applicant argues that weighted accuracy is different from weighting datapoints, which is incorrect because weighted accuracy necessarily uses weights for each datapoint. See further [0051],wherein the discriminator is trained with the probability density ratio, which produces continuous probability values and represent likelihood of dataset association – “to distinguish based on these extracted features whether the samples are from the source dataset [labeled] or the target dataset [unlabeled].”
Zhong, further explicitly teaches in [0039] “a parameter representing a probability that a sample in the source dataset appears in the target dataset is calculated”, which satisfies the limitation – “a likelihood that the datapoint is associated with the OOS dataset”.
Thus, Zhong, estimating explicitly teaches probability density ratio between source (labeled) and target datasets using discriminator trained with log-likelihood loss, which is analogous to the applicant’s own specification (see [0080] “Density estimation may, in some cases, be expensive and fickle depending on the data at hand. The discriminator technique is a technique that learns p.sub.OOS(x)/p.sub.IS(x) directly via a discriminator. This discriminator model f.sub.disc is trained to differentiate between data points from the IS and OOS distributions”; wherein the importance sampling comprises calculating, for a given model input value, a probability that the given model input value is associated with the IS dataset (p.sub.is(x)) using density estimation, calculating, for the given model input value, a probability that the given model input value is associated with the OOS dataset (p.sub.oos(x)) using density estimation” [0108]; “the importance sampling comprises density estimation of the IS dataset and the OOS dataset” [0109]). Analogously, the discriminator output, disclosed by Zhong, provides the likelihood that a datapoint is associated with target (unlabeled) dataset and that the datapoint is associated with the source (labeled) dataset. The density ration is calculated directly from these likelihoods and is used as a weight applied to each datapoint in the source dataset. Applying such weights generates a weighted subset of the source dataset. Therefore, Zhong fully discloses calculating probability based weights for datapoints and generating a weighted dataset, as required by the claim.
Applicant's remaining arguments, in regard to the presently amended claims, are addressed in the updated rejections to the claims above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure is indicated on PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to POLINA G PEACH whose telephone number is (571)270-7646. The examiner can normally be reached Monday-Friday, 9:30 - 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aleksandr Kerzhner can be reached at 571-270-1760. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/POLINA G PEACH/Primary Examiner, Art Unit 2165 April 19, 2026
Prosecution Insights