DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The present application was filed on 05/15/2023. Claims 19-43 are pending and have been examined. Claims 19, 39 and 40 are the independent claims.
Priority
Acknowledgment is made of applicant' s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The present application is a national stage entry (under 35 U.S.C. § 371) of international application no. PCT/EP2021/079215 filed on 10/21/2021, which claims foreign priority based on European Patent Application No. EP20208211 filed on 11/17/2020. The examiner notes that a certified copy of the above-noted foreign application was retrieved on 05/15/2023 as required by 37 CFR 1.55.
Claim Objections
Claims 27-29 objected to because of the following informalities:
Claim 27 should be ‘according to claim 22’ NOT ‘according to one of claim 22”.
Claim 28 should be ‘according to claim 23’ NOT ‘according to one of claim 23”.
Claim 29 should be ‘according to claim 24’ NOT ‘according to one of claim 24”.
Appropriate correction is required.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/15/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 19-43 are rejected under 35 U.S.C 112(b) or 35 U.S.C 112 (pre-AIA ), second paragraph, as failing to set forth the subject matter which the inventor or a joint inventor, or for application subject to pre-AIA 35 U.S.C 112, the application regards, as the invention.
The term “trustworthy” in claims 19, 39 and 40 is a relative term which renders the claims indefinite. The term “trustworthy” is not defined by the claims, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The Applicant’s specification merely mentions that “the provided trustworthy artificial intelligence model has a corresponding better quality” (see specification para [0010]). In particular, it is unclear what metrics or standards are used for ascertaining the requisite degree of trustworthiness for the claimed “trustworthy artificial intelligence model” in claims 19, 39 and 40. For examination purposes, “a trustworthy artificial intelligence model” is being interpreted as any improved classification model. The term “trustworthy” in claims 19, 39 and 40 is a relative term which renders the claims indefinite. The term “trustworthy” is not defined by the claims, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The Applicant’s specification merely mentions that “the provided trustworthy artificial intelligence model has a corresponding better quality” (see specification para [0010]). In particular, it is unclear what metrics or standards are used for ascertaining the requisite degree of trustworthiness for the claimed “trustworthy artificial intelligence model” in claims 19, 39 and 40. For examination purposes, “a trustworthy artificial intelligence model” is being interpreted as any improved classification model.
Claims 20-38 and 41-43, which each depend directly or indirectly from claims 19 and 40, respectively, are rejected under 35 U.S.C. 112(b) as being indefinite under the same rationale as claims 19 and 40.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 19, 20, 21, 22, 23, 24, 26, 27, 30, 31, 33, 36, 37, 40, 41 and 42 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 1, 2, 3, 4, 5, 6, 7, 8, 8, 9, 11, 12, 15, 16 and 17 of U.S. Patent No. 12,217,139 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the limitations of claims 19, 20, 21, 22, 23, 24, 26, 27, 30, 31, 33, 36, 37, 40, 41 and 42 in the present application are anticipated by claims 1, 1, 2, 3, 4, 5, 6, 7, 8, 8, 9, 11, 12, 15, 16 and 17 of U.S. Patent No. 12,217,139 B2 (please see the table below).
Regarding Claim 19, 20, 21, 22, 23, 24, 26, 27, 30, 31, 33, 36, 37, 40, 41 and 42
Instant Application No. 18/036,954 (claims filed on 05/15/2023)
Patent 12,217,139 B2 (reference application)
Claim 19:
A computer-implemented method for transforming a trained artificial intelligence model into a trustworthy artificial intelligence model, the method comprising:
providing the trained artificial intelligence model via a user interface of a webservice platform;
providing a validation data set which is based on training data of the trained artificial intelligence model;
generating generic samples by a computing component of the webservice platform based on the validation data set, the validation data set being modified by a domain-drift to generate the generic samples; and
transforming the trained artificial intelligence model by optimizing a calibration based on the generic samples.
Claim 1:
A computer-implemented method for transforming a trained artificial intelligence model into a trustworthy artificial intelligence model, comprising: providing the trained artificial intelligence model via a user interface of a webservice platform,
providing a validation data set, which is based on training data of the trained artificial intelligence model,
generating generic samples by a computing component of the webservice platform based on the validation data set, wherein for generating the generic samples the validation data set is modified by a domain-drift, and
transforming the trained artificial intelligence model by optimizing a calibration based on the generic samples, wherein by optimizing the calibration, an uncertainty-awareness is represented in a confidence-level for any of the generic samples.
Claim 20:
The method according to claim 19, wherein by optimizing the calibration, an uncertainty-awareness is represented in a confidence-level for any of the generic samples.
Claim 1:
…transforming the trained artificial intelligence model by optimizing a calibration based on the generic samples, wherein by optimizing the calibration, an uncertainty-awareness is represented in a confidence-level for any of the generic samples.
Claim 21:
The method according to claim 19, wherein the validation data set is modified according to perturbation strengths to generate the generic samples.
Claim 2:
The method according to claim 1, wherein for generating the generic samples the validation data set is modified according to perturbation strengths.
Claim 22:
The method according to claim 19, wherein said transforming comprises performing a re-training of the artificial intelligence model with applying an entropy-based loss term which encourages uncertainty-awareness.
Claim 3:
The method according to claim 1, wherein transforming comprises performing a re- training of the AI-model with applying an entropy-based loss term which encourages uncertainty- awareness.
Claim 23:
The method according to claim 22, wherein said transforming further comprises applying a calibration loss term.
Claim 4:
The method according to claim 3, wherein transforming further comprises applying a calibration loss term.
Claim 24:
The method according to claim 22, further comprising:
generating current outputs of the artificial intelligence model for the validation data set by forward propagating validation input data of the validation data set in the artificial intelligence model;
computing a categorical cross-entropy loss LCCE for the validation data set based on the current outputs and corresponding ground truth data of the validation data set;
computing a predictive entropy loss Ls by removing non-misleading evidence from the current outputs and distributing the remaining current outputs over a predetermined number C of classes;
computing a combined loss L by adding to a categorical cross-entropy loss LCCE a predictive entropy loss LS weighted with a predetermined first loss factor λS, where 0<=λS<=1;
checking whether the re-training converged to a predefined lower limit for a convergence rate;
updating weights of the artificial intelligence model based on the combined loss L and a predetermined training rate η, where 0<η<=1, in case the re-training did not converge; and
stopping the re-training of the artificial intelligence model in case the re-training converged.
Claim 5:
The method according to claim 3, comprising the steps of:
generating current outputs of the AI model for the validation data set by forward propagating validation input data of the validation data set in the AI model;
computing a categorical cross-entropy loss LCCE for the validation data set based on the current outputs and corresponding ground truth data of the validation data set;
computing a predictive entropy loss LS by removing non-misleading evidence from the current outputs and distributing the remaining current outputs over a predetermined number C of classes;
computing a combined loss L by adding to the categorical cross-entropy loss LCCE the predictive entropy loss LS weighted with a predetermined first loss factor λS, where 0<=λS<=1;
checking whether the re-training converged to a predefined lower limit for a convergence rate;
updating weights of the AI model based on the combined loss L and a predetermined training rate η, where 0<η<=1, in case the re-training did not converge; and
stopping the re-training of the AI model in case the re-training converged.
Claim 26:
The method according to claim 24, further comprising:
generating perturbed outputs of the artificial intelligence model for the generic samples by forward propagating generic input data Xadv of the generic samples in the artificial intelligence model;
computing a calibration loss Ladv as an Euclidian norm (L2 norm) of an expected calibration error ECE, which takes a weighted average over the perturbed outputs grouped in a predefined number M of equally spaced bins each having an associated average confidence and accuracy, where M>1;
checking whether the re-training converged to a predefined lower limit for a convergence rate;
first time updating weights of the artificial intelligence model based on the combined loss L and a predetermined training rate η, where 0<η<=1, in case the training did not converge;
second time updating the weights of the artificial intelligence model based on the calibration loss Ladv weighted with a predetermined second loss factor λadv, where 0<=λadv<=1, and the predetermined training rate η, in case the training did not converge; and
stopping the training of the artificial intelligence model in case the training converged.
Claim 6:
The method according to claim 5, further comprising the steps of:
generating perturbed outputs of the AI model for the generic samples by forward propagating the generic input data Xadv of the generic samples in the AI model;
computing a calibration loss Ladv as the Euclidian norm, L2 norm, of an expected calibration error ECE, which takes a weighted average over the perturbed outputs grouped in a predefined number M of equally spaced bins each having an associated average confidence and accuracy, where M>1;
checking whether the re-training converged to a predefined lower limit for a convergence rate;
first time updating weights of the AI model based on the combined loss L and a predetermined training rate η, where 0<η<=1, in case the training did not converge;
second time updating the weights of the AI model based on the calibration loss Ladv weighted with a predetermined second loss factor λadv, where 0<=λadv<=1, and the predetermined training rate η, in case the training did not converge; and
stopping the training of the AI model in case the training converged.
Claim 27:
The method according to one of claim 22, wherein the artificial intelligence model is a neural network.
Claim 7:
The method according to claim 3, wherein the artificial intelligence model is a neural network.
Claim 30:
The method according to claim 19, wherein said transforming comprises post- processing an output of the artificial intelligence model.
Claim 8:
The method according to claim 1, wherein transforming comprises post-processing an output of the AI model.
Claim 31:
The method according to claim 20, wherein said transforming comprises post- processing an output of the artificial intelligence model.
Claim 8:
The method according to claim 1, wherein transforming comprises post-processing an output of the AI model.
Claim 33:
The method according to claim 30, wherein during said step of post- processing, parameters of a monotonic function used to transform unnormalized logits are determined by optimizing a calibration metric based on the generic samples.
Claim 9:
The method according to claim 8, wherein during the step of post-processing, parameters of a monotonic function used to transform unnormalized logits are determined by optimizing a calibration metric based on the generic samples.
Claim 36:
The method according to claim 19, wherein the validation data set is a sub-set of the training data of the trained artificial intelligence model.
Claim 11:
The method according to claim 1, wherein the validation data set is a sub-set of the training data of the trained artificial intelligence model.
Claim 37:
The method according to claim 19, wherein the validation data set is generated by modifying the training data of the trained artificial intelligence model
Claim 12:
The method according to claim 1, wherein the validation data set is generated by modifying the training data of the trained artificial intelligence model.
Claim 40:
A system for transforming a trained artificial intelligence model into a trustworthy artificial intelligence model, comprising:
a user interface component to enable provision of the trained artificial intelligence model;
a memory storing the trained artificial intelligence model and user assignment information;
a computing component for generating generic samples based on a validation data set
determined based on training data of the trained artificial intelligence model and for
transforming the trained artificial intelligence model by optimizing a calibration based on the generic samples, the validation data set being modified by a domain-drift to generate the generic samples.
Claim 15:
A system for transforming a trained artificial intelligence model into a trustworthy artificial intelligence model, comprising:
a user interface component to enable provision of the trained artificial intelligence model,
a memory storing the trained artificial intelligence model and user assignment information, and
a computing component for generating generic samples based on a validation data set, wherein the validation data set is determined based on training data of the trained artificial intelligence model, wherein for generating the generic samples the validation data set is modified by a domain-drift, and for
transforming the trained artificial intelligence model by optimizing a calibration based on the generic samples, wherein by optimizing the calibration, an uncertainty-awareness is represented in a confidence-level for any of the generic samples.
Claim 41:
The system according to claim 40, wherein the user interface component is accessible via a webservice.
Claim 16:
The system according to claim 15, wherein the user interface component is accessible via a webservice.
Claim 42:
The system according to claims 40, wherein the memory and the computing component are implemented on a cloud platform.
Claim 17:
The system according to claim 15, wherein the memory and the computing component are implemented on a cloud platform.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 19-23, 27-28 and 30-43 are rejected under 35 U.S.C. 103 as being unpatentable over Marcheret (“US20130254153A1”) in view of Velagapudi (“US20210232980A1”) further in view of Kong (“Calibrated Language Model Fine-Tuning for In-and Out-of-Distribution Data”, Published on October 22 2020, part of the prior art made of record cited in applicant’s IDS filed 5/15/2023).
Claim 19.
Marcheret teaches a computer-implemented method for transforming a trained artificial intelligence model into a trustworthy artificial intelligence model1, the method comprising (Para [0046] “The inputs to method 400 comprise a classification model 410, labeled training data 420 that was used to build classification model 410, and unlabeled input 430. The output from method 400 is retrained classification model 470 that will achieve performance on the distribution of unlabeled input that is improved in at least one respect as compared with classification model 410” and Figure 4 teaches a classification model (transforming a trained artificial intelligence model) into a retrained classification model (trustworthy artificial intelligence model)):
providing the trained artificial intelligence model (Para [0087] “retraining comprises using the classification model 410 and performing further training using the reweighted labeled training data 450 to further shape the way that the classification model responds to the unlabeled input” teaches the classification models for retraining are provided);
providing a validation data set which is based on training data of the trained artificial intelligence model (Para [0046] “As mentioned above, classification model 410 was built using the labeled training data 420. The classification model 410 may have been built in any suitable manner, examples of which are described above, and classification model 410 may use any suitable classification algorithm” and Figure 4 teaches labeled training data (validation data set) of the classification model);
generating generic samples by a computing component…based on the validation data set(Para [0086] “Having determined new weight values for the labeled training data in act 440, the new weight values are associated with the labeled training data in act 450 to produce reweighted labeled training data that approximates the unlabeled test data. Reweighting of the labeled training data may be performed in any suitable manner” and Para [0137] “FIG. 10 is a block diagram of an illustrative computing device 1000 that may be used to implement any of the above-described techniques” and Figure 4 teaches labeled training data of the classification model generating reweighted labeled training data (corresponds to generic samples) by computing component);
and transforming the trained artificial intelligence model by optimizing a calibration based on the generic samples (Para [0087] “In act 460, the reweighted labeled training data is used to retrain classification model 410, thereby improving the performance of the classification model 410 for the distribution of unlabeled input 430” and Para [0085] “The above non-limiting example of calculating new weight values for the labeled training data so that it approximates the unlabeled test data to optimize the performance of a classification model may be applied in any suitable manner” also see (Paragraphs 0042-0043, 0046-0049, 0055-0058, 0075-0076, 0079-0082, 0086-0088) teaches transforming the classification model by optimizing corrected samples based on the generated samples).
Marcheret does not explicitly teach providing the trained artificial intelligence model via a user interface of a webservice platform…generating generic samples by a computing component of the webservice platform.
However, in the same field, analogous art Velagapudi teaches providing a trained artificial intelligence model via a user interface of a webservice platform (Para [0026] “FIG. 1 illustrates a One-vs.-Rest (OVR) model that includes one classifier per output represented by models 131, 132, 133, 134, 135, and 136” and Para [0030] “FIGS. 2-5 illustrate a series of example user interfaces and processes that may be performed by an example training data management application 605 illustrated in FIG. 6” and Figure 9 teaches providing a trained OVR model (artificial intelligence model) via a user interface of a training data management application 605 (webservice platform) as shown in figure 9).
generating generic samples by a computing component of the webservice platform (Para [0041] “The updated model weights 320 column illustrates what the model weight for the feature would be if the training data were modified to include the changes made via the user interface 200. The delta column 325 illustrates the difference between the updated model weight 320 and the old model weight 315” and Para [0059] “hardware, or a combination thereof, and may be implemented by the example software architecture 800 of FIG. 8 and/or the example computing device 900 illustrated in FIG. 9” teaches updated model weights by computing component of the webservice platform).
Marcheret and Velagapudi are analogous art because they are both directed to a data processing system for analyzing the impact of training data changes on a machine learning model.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Velagapudi into the disclosed invention of Marcheret.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, “The techniques for fine-tuning machine learning models discussed above provide a significant improvement over conventional approaches that require a significant investment in time and processing power to score the extensive amount of historical data used to validate the behavior of the model”, as suggested by Velagapudi (Velagapudi, Para [0073]).
Marcheret in view of Velagapudi does not explicitly teach the validation data set being modified by a domain-drift to generate the generic samples.
However, in the same field, analogous art Kong teaches the validation data set being modified by a domain-drift to generate the generic samples (Algorithm 1
PNG
media_image1.png
384
714
media_image1.png
Greyscale
teaches based on the Dx being a proper distance for features extracted fx and modified the sample, distance corresponds to domain drift).
Marcheret, Velagapudi and Kong are analogous art because they are all directed to a regularized fine-tunning method.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Kong into the disclosed invention of Marcheret in view of Velagapudi.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, the proposed regularized fine-tuning method introduces regularization “for better calibration”, thereby improving the reliability of classification outputs used for evaluation, as describe in Kong (Kong, Abstract and Page 1).
Claim 20.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 19,
Kong further teaches wherein by optimizing the calibration, an uncertainty-awareness is represented in a confidence-level for any of the generic samples (4 Experiments & Page 7 “We detect the misclassified and OOD samples by model confidence, which is the output prob ability associated with the predicted label P(X). Specifically, we setup a confidence threshold τ ∈[0,1], and take the samples with confidence below the threshold, i.e., P(X) < τ, as the misclassified or OOD samples. We can compute the detection F1 score for every τ: F1(τ), and obtain a calibration curve (F1(τ) vs. τ). Then, we set τupper as the upper bound of the confidence threshold, since a well calibrated model should provide probabilities that reflect the true likelihood and it is not reasonable to use a large τ to detect them” and Algorithm 1 teaches the on-manifold samples and off-manifold samples the generic samples) of confidence level, confidence level comprised misclassified (un uncertainty awareness) by optimizing the calibrated model).
Marcheret, Velagapudi and Kong are analogous art because they are all directed to a regularized fine-tunning method.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Kong into the disclosed invention of Marcheret in view of Velagapudi.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, the proposed regularized fine-tuning method introduces regularization “for better calibration”, thereby improving the reliability of classification outputs used for evaluation, as describe in Kong (Kong, Abstract and Page 1).
Claim 21.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 19,
Kong further teaches wherein the validation data set is modified according to perturbation strengths to generate the generic samples (Section 1 Introduction & Page 3 “(2) Off-manifold regularization: We generate off-manifold samples by adding relatively large perturbations along the directions that point outward the data manifold” and 3.2 Off-manifold Regularization & Page 6 “The off-manifold regularizer, R2, encourages the model to yield low confidence outputs for samples outside the data manifold, and thus mitigates the over-confidence issue for out-of-distribution (OOD) data. Specifically, given a training sample (x,y), we generate an off-manifold pseudo sample x ∗” teaches the giving training sample modified according to perturbation to generate samples).
Marcheret, Velagapudi and Kong are analogous art because they are all directed to a regularized fine-tunning method.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Kong into the disclosed invention of Marcheret in view of Velagapudi.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, the proposed regularized fine-tuning method introduces regularization “for better calibration”, thereby improving the reliability of classification outputs used for evaluation, as describe in Kong (Kong, Abstract and Page 1).
Claim 22.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 19,
Kong further teaches wherein said transforming comprises performing a re-training of the artificial intelligence model with applying an entropy-based loss term which encourages uncertainty-awareness (Section 3.2 Off-manifold Regularization and Page 7 “where Soff denotes the set of all generated off-manifold samples, and H(·) denotes the entropy of the probability simplex” and 2 Preliminaries and Page 4 “A well-calibrated model is expected to produce an output with high uncertainty for such out-of-distribution (OOD) data” and algorithm 1 teaches transforming comprises performing retraining of model with applying an entropy based lose term for the probability simplex).
Marcheret, Velagapudi and Kong are analogous art because they are all directed to a regularized fine-tunning method.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Kong into the disclosed invention of Marcheret in view of Velagapudi.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, the proposed regularized fine-tuning method introduces regularization “for better calibration”, thereby improving the reliability of classification outputs used for evaluation, as describe in Kong (Kong, Abstract and Page 1).
Claim 23.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 22,
Kong further teaches wherein said transforming further comprises applying a calibration loss term (2 Preliminaries & Page 4 “The calibration error of the predictive model for a given confidence p ∈ (0,1) is defined as: Ep =|P(Y(X)=Y(X)|P(X)=p)−p|. (1)” teaches calibration error).
Marcheret, Velagapudi and Kong are analogous art because they are all directed to a regularized fine-tunning method.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Kong into the disclosed invention of Marcheret in view of Velagapudi.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, the proposed regularized fine-tuning method introduces regularization “for better calibration”, thereby improving the reliability of classification outputs used for evaluation, as describe in Kong (Kong, Abstract and Page 1).
Claim 27.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to one of claim 22,
Marcheret further teaches wherein the artificial intelligence model is a neural network (Para [0024] “The classification model may be retrained in an unsupervised fashion to achieve a desired level of performance (i.e., without requiring supervised labeling of the test data), and the performance of the retrained classification model on the unlabeled test data may closely match that of the model on the reweighted labeled training data” teaches the classification model may be retrained in an unsupervised (corresponds to the artificial intelligence model)).
Claim 28.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to one of claim 23,
Marcheret further teaches wherein the artificial intelligence model is a neural network (Para [0024] “The classification model may be retrained in an unsupervised fashion to achieve a desired level of performance (i.e., without requiring supervised labeling of the test data), and the performance of the retrained classification model on the unlabeled test data may closely match that of the model on the reweighted labeled training data” teaches the classification model may be retrained in an unsupervised (corresponds to the artificial intelligence model)).
Claim 30.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 19,
Marcheret further teaches wherein said transforming comprises post-processing an output of the artificial intelligence model (Para [0024] “The classification model may be retrained in an unsupervised fashion to achieve a desired level of performance (i.e., without requiring supervised labeling of the test data), and the performance of the retrained classification model on the unlabeled test data may closely match that of the model on the reweighted labeled training data” and Para [0030] “some embodiments relate to the retraining of a classification model. A classification model may comprise any suitable classification algorithm (also known as a ‘classifier’), non limiting examples of which include logistic regression, support vector machine (SVM) and an exponential family model” teaches the classification model may be retrained in an unsupervised (corresponds to the artificial intelligence model)).
Claim 31.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 20,
Marcheret further teaches wherein said transforming comprises post- processing an output of the artificial intelligence model (Para [0024] “The classification model may be retrained in an unsupervised fashion to achieve a desired level of performance (i.e., without requiring supervised labeling of the test data), and the performance of the retrained classification model on the unlabeled test data may closely match that of the model on the reweighted labeled training data” and Para [0030] “some embodiments relate to the retraining of a classification model. A classification model may comprise any suitable classification algorithm (also known as a ‘classifier’), non limiting examples of which include logistic regression, support vector machine (SVM) and an exponential family model” teaches the classification model may be retrained in an unsupervised (corresponds to the artificial intelligence model)).
Claim 32.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 21,
Marcheret further teaches wherein said transforming comprises post- processing an output of the artificial intelligence model (Para [0024] “The classification model may be retrained in an unsupervised fashion to achieve a desired level of performance (i.e., without requiring supervised labeling of the test data), and the performance of the retrained classification model on the unlabeled test data may closely match that of the model on the reweighted labeled training data” and Para [0030] “some embodiments relate to the retraining of a classification model. A classification model may comprise any suitable classification algorithm (also known as a ‘classifier’), non limiting examples of which include logistic regression, support vector machine (SVM) and an exponential family model” teaches the classification model may be retrained in an unsupervised (corresponds to the artificial intelligence model)).
Claim 33.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 30,
Kong further teaches wherein during said step of post- processing, parameters of a monotonic function used to transform unnormalized logits are determined by optimizing a calibration metric based on the generic samples (3.1 On-manifold Regularization & Page 5 “The on-manifold regularizer Ron exploits the interpolation of training data within the data manifold to improve the in-distribution calibration. Specifically, given two training samples (x,y) and (x,y) and the feature extraction layers f , we generate an on-manifold pseudo sample (x,y)” and 3.2 Off-manifold Regularization & Page 6 “The off-manifold regularizer, R2, encourages the model to yield low confidence outputs for samples outside the data manifold, and thus mitigates the over-confidence issue for out-of-distribution (OOD) data” and Algorithm 1 and Figure 3 teaches increasing (monotonic) function used to transform unnormalized parameters are determined by optimizing a calibration metric based on the generic samples).
Marcheret, Velagapudi and Kong are analogous art because they are all directed to a regularized fine-tunning method.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Kong into the disclosed invention of Marcheret in view of Velagapudi.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, the proposed regularized fine-tuning method introduces regularization “for better calibration”, thereby improving the reliability of classification outputs used for evaluation, as describe in Kong (Kong, Abstract and Page 1).
Claim 34.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 30,
Marcheret further teaches wherein the artificial intelligence model is a classifier comprising one of a deep neural network, gradient boosted decision tree, xgboost, support vector machine, random forest and neural network (Para [0030] “A classification model may comprise any suitable classification algorithm (also known as a ‘classifier’), non limiting examples of which include logistic regression, support vector machine (SVM) and an exponential family model” and Para [0084] “the conjugate gradient method” teaches the artificial intelligence model is classifier comprising support vector machine and gradient method).
Claim 35.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 33,
Marcheret further teaches wherein the artificial intelligence model is a classifier comprising one of a deep neural network, gradient boosted decision tree, xgboost, support vector machine, random forest and neural network (Para [0030] “A classification model may comprise any suitable classification algorithm (also known as a ‘classifier’), non limiting examples of which include logistic regression, support vector machine (SVM) and an exponential family model” and Para [0084] “the conjugate gradient method” teaches the artificial intelligence model is classifier comprising support vector machine and gradient method).
Claim 36.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 19,
Marcheret further teaches wherein the validation data set is a sub-set of the training data of the trained artificial intelligence model (Para [0076] “Application of a new set of weights for the labeled training data may be implemented in any suitable manner. For example, only a subset of the labeled training data may be reweighted such that part of the data set is unchanged by the reweighting, or a complete recalculation of all weights associated with the labeled training data may be performed” teaches a subset of the labeled training data (validation data set) of the model).
Claim 37.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 19,
Marcheret further teaches wherein the validation data set is generated by modifying the training data of the trained artificial intelligence model (Para [0076] “Application of a new set of weights for the labeled training data may be implemented in any suitable manner. For example, only a subset of the labeled training data may be reweighted such that part of the data set is unchanged by the reweighting, or a complete recalculation of all weights associated with the labeled training data may be performed” teaches a subset of the labeled training data (validation data set) is generated by the modifying the labeled training data of the model).
Claim 38.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the method according to claim 19,
Velagapudi further teaches wherein the transformed artificial intelligence model is provided via the user interface of the webservice platform as a downloadable file (Para [0033] “The user interface 200 of the training data management application 605 provides means for a user to search for historical data by class and to update the historical data associated with the class to be included in the training data used to train the machine learning model. The user interface 200 includes a search field 205 in which a user may enter class identifier, class description, or other information that may be used to identify the class. The user interface 200 may implemented as part of a training data management application 605 implemented locally on the data processing system of the user or may be an interface to a cloud-based implementation of the application for managing training data for machine learning models. In some implementations, the user interface may include user interface component(s) suitable for specifying or selecting a location in memory of the data processing system or a network location where historical data and/or training data for the machine learning model is located” teaches the transformed artificial intelligence model is provided via the user interface of the search for historical data (webservice platform) as a downloadable file).
Marcheret and Velagapudi are analogous art because they are both directed to a data processing system for analyzing the impact of training data changes on a machine learning model.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Velagapudi into the disclosed invention of Marcheret.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, “The techniques for fine-tuning machine learning models discussed above provide a significant improvement over conventional approaches that require a significant investment in time and processing power to score the extensive amount of historical data used to validate the behavior of the model”, as suggested by Velagapudi (Velagapudi, Para [0073]).
Claim 39.
With respect to independent claim 39, claim 39 is substantially similar to claim 19 and therefore is rejected on the same ground as claim 19, discussed above. In particular, claim 39 is a computer program product that perform operations of claim 19.
Marcheret further teaches a non-transitory computer program product encoded with program instructions which, when executed by a computing component, cause the computing component to transform a trained artificial intelligence model into a trustworthy artificial intelligence model, the program instructions comprising (Para [0046] “The inputs to method 400 comprise a classification model 410, labeled training data 420 that was used to build classification model 410, and unlabeled input 430. The output from method 400 is retrained classification model 470 that will achieve performance on the distribution of unlabeled input that is improved in at least one respect as compared with classification model 410” and Figure 4 and Para [0137] “Computing device 1000 may include one or more processors 1001 and one or more tangible, non-transitory computer-readable storage media (e.g., memory 1003)” teaches a classification model (transforming a trained artificial intelligence model) into a retrained classification model (trustworthy artificial intelligence model)).
Claim 40.
Marcheret teaches a system for transforming a trained artificial intelligence model into a trustworthy artificial intelligence model2, comprising (Para [0046] “The inputs to method 400 comprise a classification model 410, labeled training data 420 that was used to build classification model 410, and unlabeled input 430. The output from method 400 is retrained classification model 470 that will achieve performance on the distribution of unlabeled input that is improved in at least one respect as compared with classification model 410” and Figure 4 teaches a classification model (transforming a trained artificial intelligence model) into a retrained classification model (trustworthy artificial intelligence model)):
…the trained artificial intelligence model (Para [0087] “retraining comprises using the classification model 410 and performing further training using the reweighted labeled training data 450 to further shape the way that the classification model responds to the unlabeled input” teaches the classification models for retraining are provided);
a memory storing the trained artificial intelligence model and user assignment information (Para [0133] “Plot 900 may be provided to a user in any suitable way (e.g., on a display screen via a computing device, via a printout, etc.). The data represented by plot 900 may be stored in one or more storage devices of a computing system (e.g., a disk drive or memory)” and Para [0135] “ a classification model trained on the transformed labeled training data may perform better at classifying the unlabeled data than a classification model trained on the original labeled training data” teaches memory storing the trained model and plot 900 (corresponds to user assignment information));
a computing component for generating generic samples based on a validation data set determined based on training data of the trained artificial intelligence model (Para [0046] “As mentioned above, classification model 410 was built using the labeled training data 420. The classification model 410 may have been built in any suitable manner, examples of which are described above, and classification model 410 may use any suitable classification algorithm” and Figure 4 teaches subset of labeled training data (validation data set) of the classification model and Para [0086] “Having determined new weight values for the labeled training data in act 440, the new weight values are associated with the labeled training data in act 450 to produce reweighted labeled training data that approximates the unlabeled test data. Reweighting of the labeled training data may be performed in any suitable manner” and Para [0137] “FIG. 10 is a block diagram of an illustrative computing device 1000 that may be used to implement any of the above-described techniques” and Figure 4 teaches labeled training data of the classification model generate reweighted labeled training data by computing component)
and for transforming the trained artificial intelligence model by optimizing a calibration based on the generic samples (Para [0087] “In act 460, the reweighted labeled training data is used to retrain classification model 410, thereby improving the performance of the classification model 410 for the distribution of unlabeled input 430” and Para [0085] “The above non-limiting example of calculating new weight values for the labeled training data so that it approximates the unlabeled test data to optimize the performance of a classification model may be applied in any suitable manner” also see (Paragraphs 0042-0043, 0046-0049, 0055-0058, 0075-0076, 0079-0082, 0086-0088) teaches transforming the classification model by optimizing corrected samples based on the generated samples).
Marcheret does not explicitly teach a user interface component to enable provision of the trained artificial intelligence model.
However, in the same field, analogous art Velagapudi teaches a user interface component to enable provision of the trained artificial intelligence model (Para [0026] “FIG. 1 illustrates a One-vs.-Rest (OVR) model that includes one classifier per output represented by models 131, 132, 133, 134, 135, and 136” and Para [0030] “FIGS. 2-5 illustrate a series of example user interfaces and processes that may be performed by an example training data management application 605 illustrated in FIG. 6” and Figure 9 teaches providing a trained OVR model (artificial intelligence model) via a user interface of a training data management application 605 (webservice platform) as shown in figure 9).
Marcheret and Velagapudi are analogous art because they are both directed to a data processing system for analyzing the impact of training data changes on a machine learning model.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Velagapudi into the disclosed invention of Marcheret.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, “The techniques for fine-tuning machine learning models discussed above provide a significant improvement over conventional approaches that require a significant investment in time and processing power to score the extensive amount of historical data used to validate the behavior of the model”, as suggested by Velagapudi (Velagapudi, Para [0073]).
Marcheret in view of Velagapudi does not explicitly teach the validation data set being modified by a domain-drift to generate the generic samples.
However, in the same field, analogous art Kong teaches the validation data set being modified by a domain-drift to generate the generic samples (Algorithm 1
PNG
media_image1.png
384
714
media_image1.png
Greyscale
teaches based on the Dx being a proper distance for features extracted fx and modified the sample, distance corresponds to domain drift).
Marcheret, Velagapudi and Kong are analogous art because they are all directed to a regularized fine-tunning method.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Kong into the disclosed invention of Marcheret in view of Velagapudi.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, the proposed regularized fine-tuning method introduces regularization “for better calibration”, thereby improving the reliability of classification outputs used for evaluation, as describe in Kong (Kong, Abstract and Page 1).
Claim 41.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the system according to claim 40,
Velagapudi further teaches wherein the user interface component is accessible via a webservice Para [0026] “FIG. 1 illustrates a One-vs.-Rest (OVR) model that includes one classifier per output represented by models 131, 132, 133, 134, 135, and 136” and Para [0030] “FIGS. 2-5 illustrate a series of example user interfaces and processes that may be performed by an example training data management application 605 illustrated in FIG. 6” and Figure 9 teaches providing a trained OVR model (artificial intelligence model) via a user interface of a webservice platform as shown in figure 9).
Marcheret and Velagapudi are analogous art because they are both directed to a data processing system for analyzing the impact of training data changes on a machine learning model.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Velagapudi into the disclosed invention of Marcheret.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, “The techniques for fine-tuning machine learning models discussed above provide a significant improvement over conventional approaches that require a significant investment in time and processing power to score the extensive amount of historical data used to validate the behavior of the model”, as suggested by Velagapudi (Velagapudi, Para [0073]).
Claim 42.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the system according to claim 40,
Velagapudi further teaches wherein the memory and the computing component are implemented on a cloud platform (Para [0089] “the machine 900 to perform and one or more of the features described herein. Accordingly, a “machine-readable medium” may refer to a single storage device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices” teaches wherein the memory and the cloud based storage systems).
Marcheret and Velagapudi are analogous art because they are both directed to a data processing system for analyzing the impact of training data changes on a machine learning model.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Velagapudi into the disclosed invention of Marcheret.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, “The techniques for fine-tuning machine learning models discussed above provide a significant improvement over conventional approaches that require a significant investment in time and processing power to score the extensive amount of historical data used to validate the behavior of the model”, as suggested by Velagapudi (Velagapudi, Para [0073]).
Claim 43.
As discussed above, Marcheret in view of Velagapudi further in view of Kong teach the system according to claim 41,
Velagapudi further teaches wherein the memory and the computing component are implemented on a cloud platform (Para [0089] “the machine 900 to perform and one or more of the features described herein. Accordingly, a “machine-readable medium” may refer to a single storage device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices” teaches wherein the memory and the cloud-based storage systems).
Marcheret and Velagapudi are analogous art because they are both directed to a data processing system for analyzing the impact of training data changes on a machine learning model.
It would have been obvious for one of ordinary skill in the arts before the effective filing date of the claimed invention to incorporate the limitation(s) above as taught by Velagapudi into the disclosed invention of Marcheret.
One of ordinary skill in the arts would have been motivated to make this modification because of the following, “The techniques for fine-tuning machine learning models discussed above provide a significant improvement over conventional approaches that require a significant investment in time and processing power to score the extensive amount of historical data used to validate the behavior of the model”, as suggested by Velagapudi (Velagapudi, Para [0073]).
Conclusion
6. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Lokesha Patel whose telephone number is (571)272-6267. The examiner can normally be reached 8 AM - 4 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LOKESHA PATEL/ Examiner, Art Unit 2125
/KAMRAN AFSHAR/ Supervisory Patent Examiner, Art Unit 2125
1 As indicated above in the section 112(b) rejection of this claim, “a trustworthy artificial intelligence model” has been interpreted as any improved classification model
2 As indicated above in the section 112(b) rejection of this claim, “a trustworthy artificial intelligence model” has been interpreted as any improved classification model