DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The following action is in response to the original filing of 10/09/2023.
Claims 1-20 are pending and have been considered below.
Drawings
The drawings of 10/09/2023 are objected to under 37 CFR 1.83(a) because they contain details and features which are blurry and hard to read/comprehend (see at least Fig. 2-4).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as "amended." If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because it contains an embedded hyperlink and/or other form of browser-executable code. Applicant is required to delete the embedded hyperlink and/or other form of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
The use of trade name or a mark used in commerce, has been noted in this application (ex. specification pp. 57). The terms should be accompanied by the generic terminology; furthermore the terms should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the terms.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 7, 10, 15, 17 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 5 and 15, each claim recites the limitation "the entropies". There is insufficient antecedent basis for this limitation in the claim.
Regarding claims 7 and 17, each claim recites the limitation "the distance". There is insufficient antecedent basis for this limitation in the claim.
Regarding claims 10 and 20, each claim recites the phrase “is presumed to”. Use of the term “presumed” renders the claim indefinite it is unclear whether the limitations following the phrase are part of the claimed invention. See MPEP § 2173.05(d).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Regarding claims 1 and 11:
Step 1, MPEP 2106.03:
These limitations have been determined, under Step 1, to be statutory categories of invention:
A method [..] (claim 1)
A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors [..] (claim 11)
Step 2A Prong One MPEP 2106.04, 2106.04(a):
These limitations represent, under Step 2A Prong One, mental processes such as concepts that can be practically performed in the human mind, or by a human using pen and paper as a physical aid, including observations, evaluations, judgments and opinions, MPEP 2106.04(a)(2)(III), for example a a user can ascertain certainty/confidence of a model based on each sample and evaluate an overall certainty/confidence of the model, then determine a score based on the certainties/confidences and data distributions in order to decide a drift detection policy for the model:
[..] determining a certainty of .. model on a per-sample basis; [..]
[..] determining an overall certainty of the .. model; [..]
[..] determining a sensitivity score for the model based on the overall certainty of the .. model, a .. distribution, and a .. distribution; [..]
[..] executing a drift detection policy on the .. model, wherein the drift detection policy is based on the sensitivity score of the .. model. [..]
These limitations represent, under Step 2A Prong One, mathematical concepts such as mathematical relationships, mathematical formulas or equations, or mathematical calculations, MPEP 2106.04(a)(2)(I):
[..] determining a sensitivity score for the model based on ., a maximally certain distribution, and a maximally uncertain distribution; [..]
Step 2A Prong Two, MPEP 2106.04(d):
These limitations represent, under Step 2A Prong Two, mere instructions to implement the abstract idea using generic computing tools, MPEP 2106.05(f):
[..] A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: [..] (claim 11)
[..] a machine learning model [..]
Step 2B, MPEP 2106.05:
These limitations are considered, under Step 2B, insignificant extra-solution activity as being recited at a high level of generality, MPEP 2106.05(d):
[..] A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: [..] (claim 11)
[..] a machine learning model [..]
Regarding dependent claims 2-7, 9-10, 12-17 and 19-20, these dependent claim further recites limitations for determining and normalizing entropy outputs (claims 2-4, 12-14), determining and defining distributions (claims 5-7, 15-17) and determining frequency/resilience as a score approaches a limit (claims 9-10, 19-20). The analysis incorporates the Step analysis of its respective parent. These limitations represent, under Step 2A Prong One, mathematical concepts such as mathematical relationships, mathematical formulas or equations, or mathematical calculations, MPEP 2106.04(a)(2)(I). Under Steps 2A Prong Two and Step 2B, all respective limitations are part of the abstract idea.
Regarding dependent claims 8 and 18, these dependent claim further additionally recite limitations that specify a frequency of the policy (claims 8, 18). The analysis incorporates the Step analysis of its respective parent. The additional limitations represent, under Step 2A Prong Two, mere instructions to apply at a high level of generality (MPEP 2106.05); under Step 2B, these additional limitations are considered mere instructions to apply to obtain a solution/outcome (MPEP 2106.05(f)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Upadhyay, US 2024/0206821 A1 effective filing of 12/23/2022 in view of Sadiq, US 2024/0330254 A1 effective filing 03/27/2023.
Regarding claim 1, Upadhyay discloses a method comprising:
determining a certainty of a machine learning model on a per-sample basis (pp. 46: calculate uncertainty metric, pp. 94: calculate uncertainty for each sample);
determining an overall certainty of the machine learning model (pp. 47-48: calibrate uncertainty metric for model);
determining a sensitivity score for the model based on the overall certainty of the machine learning model (pp. 50: comparing uncertainty metric to threshold to determine trustworthiness), a maximally certain distribution, and a maximally uncertain distribution (pp. 48-49: between predicted probabilities, maximally uncertain, and true probabilities, maximally certain); and
executing a policy on the machine learning model, wherein the policy is based on the sensitivity score of the machine learning model (pp. 52: executing feedback and retraining based on uncertainty metric threshold comparison).
While Upadhyay discloses drift detection and correction (pp. 53: baseline drift correction), Upadhyay fails to disclose wherein the executed policy is a drift detection policy.
Sadiq discloses methods for drift detection based on gathered data (pp. 9-10), an analogous art. In particular, Sadiq discloses executing a drift detection policy based on performance metric computations (pp. 45: run drift detection polices at certain frequencies based on performance metric). Accordingly, it would have been obvious to one having ordinary skill in the art and the teachings of Upadhyay and Sadiq before them before the effective filing of the claimed invention to combine the execution of a drift detection policy based on determined metrics, as taught by Sadiq, with the execution of a policy based on the determined metrics of Upadhyay. One would have been motivated to make this combination in order to provide more efficient operations reducing computational resources and/or time to compute, such as taking as few data scans as possible, as suggested by Sadiq (pp. 7, pp. 45).
Regarding claim 2, Upadhyay and Sadiq disclose the method of claim 1, and Upadhyay further discloses wherein determining the certainty of the machine learning model on the per-sample basis includes determining an entropy of an output of the machine learning model for a sample (pp. 46: entropy, pp. 94: per sample).
Regarding claim 3, Upadhyay and Sadiq disclose the method of claim 2, and Upadhyay further comprising normalizing the entropy of the output (pp. normalize uncertainty metric).
Regarding claim 4, Upadhyay and Sadiq disclose the method of claim 3, and Upadhyay further discloses determining and normalizing a second entropy that is based on two top values included in the output.
Regarding claim 5, Upadhyay and Sadiq disclose the method of claim 1, and Upadhyay further discloses wherein determining the overall certainty includes defining a first distribution of the entropies of each sample in a validation dataset (pp. 47-48: validation dataset used for calibration overall certainty).
Regarding claim 6, Upadhyay and Sadiq disclose the method of claim 5, and Upadhyay further discloses wherein determining the sensitivity score includes determining a distance between the first distribution and the maximally certain distribution and the maximally uncertain distribution (pp. 48-49: minimize difference between predicted probability distribution and true probabilities distribution, pp. 58: distance metrics used in determining semantic ‘closeness’, pp. 81, Fig. 6: clustering algorithms for detecting closeness between distributions).
Regarding claim 7, Upadhyay and Sadiq disclose the method of claim 1, and Sadiq further discloses wherein the distance is a Wasserstein distance (pp. 47: Wasserstein matrix used for statistical distances).
Regarding claim 8, Upadhyay and Sadiq disclose the method of claim 6, and Sadiq further discloses wherein the drift detection policy specifies a frequency at which the drift detection policy is performed on the machine learning model (pp. 45).
Regarding claim 9, Upadhyay and Sadiq disclose the method of claim 9, and Upadhyay further discloses wherein the frequency of the drift detection policy increases as the sensitivity score approaches 1 (pp. 50: threshold set at 0.93).
Regarding claim 10, Upadhyay and Sadiq disclose the method of claim 9, and Upadhyay further discloses wherein a resiliency of the machine learning model to data drift and/or context drift is presumed to decrease as the sensitivity score approaches 1 (pp. 50: threshold set at 0.93).
Regarding claims 11-20, claims 11-20 recite limitations similar to claims 1-10, respectively, and are similarly rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Soman; Sumit et al.
US 20250053618 A1
NODE AND METHODS PERFORMED THEREBY FOR HANDLING DRIFT IN DATA
Byun; Seok-Yong et al.
US 20240370731 A1
UNSUPERVISED MODEL DRIFT ESTIMATION SYSTEM FOR DATASET SHIFT DETECTION AND MODEL SELECTION
Srinath; Srikar et al.
US 20240370776 A1
TECHNIQUES FOR MONITORING MACHINE LEARNING MODEL PERFORMANCE
Swaroop; Prem et al.
US 20240211798 A1
INDUSTRIAL MONITORING PLATFORM
Gottin V et al.
US 20230244957 A1
SYSTEM FOR MANAGING MACHINE LEARNING MODELS USING EDGE CONCEPT DRIFT DURATION PREDICTION FOR NODES IN EDGE ENVIRONMENT, HAS PROCESSING DEVICE FOR PREDICTING DRIFT PERIOD DURATION FOR DATASET USING MACHINE LEARNING-BASED DRIFT MODEL THAT IS TRAINED BASED ON CONFIDENCE VALUES
Lopatecki; Jason et al.
US 20230186144 A1
DATA DRIFT IMPACT IN A MACHINE LEARNING MODEL
VAN Bruwaene DAVID et al.
WO 2023044555 A1
SYSTEM AND METHOD FOR ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING MODEL VALIDATION
Sundar; Bharath Ram et al.
US 20220246457 A1
DETECTING OUTLIERS AT A MANUFACTURING SYSTEM USING MACHINE LEARNING
Badawy; Mohamed M. et al.
US 11295241 B1
SYSTEM AND METHOD FOR INCREMENTAL TRAINING OF MACHINE LEARNING MODELS IN ARTIFICIAL INTELLIGENCE SYSTEMS, INCLUDING INCREMENTAL TRAINING USING ANALYSIS OF NETWORK IDENTITY GRAPHS
Tamir; Ganir et al.
US 20220036201 A1
REAL DRIFT DETECTOR ON PARTIAL LABELED DATA IN DATA STREAMS
Farahani; Farhad et al.
US 20080183649 A1
APPARATUS, METHOD AND SYSTEM FOR MAXIMUM ENTROPY MODELING FOR UNCERTAIN OBSERVATIONS
Trick, Susanne, et al. "Multimodal uncertainty reduction for intention recognition in human-robot interaction." 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.
Shames, Iman, and Farhad Farokhi. "Online stochastic convex optimization: Wasserstein distance variation." arXiv preprint arXiv:2006.01397 (2020).
Gireesh, Elakkat D., and Varadaraj P. Gurupur. "Information entropy measures for evaluation of reliability of deep neural network results." Entropy 25.4 (2023): 573.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW L TANK whose telephone number is (571)270-1692. The examiner can normally be reached Monday-Thursday 9a-6p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW L TANK/Primary Examiner, Art Unit 2141
Prosecution Insights