Detailed Action
This action is in responsive to the application filed 03/29/2024, in which:
Claims 1, 8, and 15 are the independent claims.
Claims 1-20 are currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 03/29/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
The abstract of the disclosure is objected to because objected to because of the words in length (over 150 words). A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claims 1-20 are objected to because of the following informalities:
Claims 1, 8, and 15 recite the term at within “… determining a first accuracy score of a trained machine learning model at determining anomalies in a first set of data items at least in part by” which appears to be grammatically incorrect. Thus, the dependent claims are also objected. The examiner is interpreting the at term as for. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 7-12, and 14-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 1 further recites the method comprising of:
determining a first accuracy score of a trained machine learning model at … (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
… determining anomalies in a first set of data items at least in part by: (a human being can mentally apply evaluation to determine anomalies in a specific set of data)
comparing the first set of outputs of the trained machine learning model to a first labeled version of the first set of data items to determine a first set of incorrectly labeled outputs (a human being can mentally apply evaluation to compare specific sets of outputs to specific data items to determine incorrectly labeled outputs)
determining a contiguous anomaly score value region that includes: a threshold portion of … (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
determining an updated accuracy score … (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
… determining anomalies in a superset of data items comprising the first set of data items and the second set of data items at least in part by: (a human being can mentally apply evaluation to determine anomalies in a specific set of data)
selecting a second subset of data items within the contiguous anomaly score value region, wherein the second subset of data items has fewer items than the second set of data items (a human being can mentally apply evaluation and make a judgment to select data within a specific region with specific requirements)
clustering the second subset of data items into a plurality of clusters based at least in part on one or more feature values of the second subset of data items (a human being can mentally apply evaluation to cluster specific data items into multiple clusters with specific requirements)
selecting a third subset of data items from the second subset of data items such that: the third subset has fewer items than the second subset, and the third subset has one or more data items in each cluster of the plurality of clusters (a human being can mentally apply evaluation and make a judgment to select data with specific requirements)
determining a second accuracy score … (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
… comparing … a third subset of labeled outputs of the third subset of data items to the labeled feedback (a human being can mentally apply evaluation to compare specific sets of outputs to labeled feedback)
combining the first accuracy score and the second accuracy score (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
based at least in part on the updated accuracy score, determining whether the trained machine learning model satisfies one or more conditions for retraining the trained machine learning model (a human being can mentally apply evaluation to determine whether a model satisfies specific conditions for retraining based on a specific accuracy score)
based at least in part on determining that the trained machine learning model satisfies the one or more conditions … (a human being can mentally apply evaluation to determine a model satisfies specific conditions)
Claim 1 thus recites an abstract idea (that falls into the “mental processes” and “mathematical concepts” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the additional elements consist of:
A computer-implemented method comprising: (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
providing a first unlabeled version of the first set of data items to the trained machine learning model as a first set of inputs … (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
… to generate a first set of outputs of the trained machine learning model … (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
… wherein the first set of outputs is labeled with a first set of anomaly scores (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
… the first set of incorrectly labeled outputs, one or more outputs of the first set of outputs labeled as anomalous, and one or more outputs of the first set of outputs labeled as not anomalous (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
receiving a second set of data items that have not been labeled (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
providing the second set of data items to the trained machine learning model as a second set of inputs … (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
… to generate a second set of outputs of the trained machine learning model (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
… wherein the second set of outputs is labeled with a second set of anomaly scores (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
collecting labeled feedback for the third subset of data items (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
… at least in part by … from the trained machine learning model … (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
… initiating retraining of the trained machine learning model (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements recited, alone or in combination, do not provide significantly more than the abstract idea itself. Additional elements a, d-e, and i are only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Additional elements b, f-g, and j fall within MPEP 2106.05(d) as well-understood, routine and conventional activities of receiving or transmitting data over a network (MPEP 2106.05(d)(II): buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)). Additional elements c, h, and k-l are merely applying the abstract idea on a computer (MPEP 2106.05(f)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claim 2:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 2 recites the method of Claim 1. Claim 1 is a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 2 does not recite any additional abstract ideas and only inherits the abstract ideas from Claim 1. Claim 2 thus recites an abstract idea (that falls into the “mental processes” and “mathematical concepts” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the sole additional element recited consists of sending a notification to an administrator of the trained machine learning model, wherein the notification provides a summary comprising the updated accuracy score and a time for the retraining (which is insignificant extra-solution activity of data display or output, by MPEP 2106.05(g)).
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the new sole additional element recited, alone or in combination, does not provide significantly more than the abstract idea itself. The additional element falls within MPEP 2106.05(d) as well-understood, routine and conventional activities of presenting offers and gathering statistics (MPEP 2106.05(d)(II)). Thus, the claim is subject-matter ineligible.
Regarding Claim 3:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 3 recites the method of Claim 1. Claim 1 is a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 3 does not recite any additional abstract ideas and only inherits the abstract ideas from Claim 1. Claim 3 thus recites an abstract idea (that falls into the “mental processes” and “mathematical concepts” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the additional elements consist of:
… scheduling the retraining … (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
based at least in part on two or more different frequencies … (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
… by which data items are provided to the trained machine learning model over two or more windows of time (which is insignificant extra-solution activity of data display or output, by MPEP 2106.05(g))
wherein the retraining is scheduled for a particular window of time of the two or more windows of time (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements recited, alone or in combination, do not provide significantly more than the abstract idea itself. Additional element a is merely applying the abstract idea on a computer (MPEP 2106.05(f)) which cannot provide significantly more. Additional elements b and d are only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Additional element c falls within MPEP 2106.05(d) as well-understood, routine and conventional activities of receiving or transmitting data over a network (MPEP 2106.05(d)(II): buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)). Thus, the claim is subject-matter ineligible.
Regarding Claim 4:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 4 recites the method of Claim 1. Claim 1 is a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 4 further recites the method comprising of:
… predict multivariate anomalies in a physical system (a human being can mentally apply evaluation to predict an unusual pattern or datapoint within a specific system/environment)
determining a first accuracy score of a trained machine learning model at … (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
Claim 4 thus recites an abstract idea (that falls into the “mental processes” and “mathematical concepts” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the additional elements consist of:
wherein the second subset of data items comprise sensor values from sensors measuring physical properties of the physical system (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
wherein the sensors are separately identified and tracked in an anomaly detection platform (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
wherein the sensors stream the second subset of data items into the anomaly detection platform using connections that provide sensor-identifying information (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements recited, alone or in combination, do not provide significantly more than the abstract idea itself. Additional elements a-b are only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Additional element c falls within MPEP 2106.05(d) as well-understood, routine and conventional activities of receiving or transmitting data over a network (MPEP 2106.05(d)(II): buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)). Thus, the claim is subject-matter ineligible.
Regarding Claim 5:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 5 recites the method of Claim 1. Claim 1 is a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 5 further recites the method comprising of:
… randomly selecting a unique data item from the second subset of data items (a human being can mentally apply evaluation and make a judgment to select data within a specific dataset)
assigning the unique data item to a particular cluster of the plurality of clusters (a human being can mentally apply evaluation and make a judgment to assign specific data to a particular cluster)
re-performing said randomly selecting if adding the unique data item to the third subset of data items would result in an over-representation of the particular cluster (a human being can mentally apply evaluation and make a judgment to re-perform selecting data within a specific dataset)
Claim 5 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited.
Regarding Claim 7:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 7 recites the method of Claim 1. Claim 1 is a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 7 further recites the method comprising of … tuning one or more hyperparameters of the trained machine learning model based at least in part on the third subset of labeled outputs (a human being can mentally apply evaluation and make a judgment to tune hyperparameters of a trained ML model based on specific data). Claim 7 thus recites an abstract idea (that falls into the “mental processes” and “mathematical concepts” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the sole additional element recited consists of (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h)).
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the new sole additional element recited, alone or in combination, does not provide significantly more than the abstract idea itself. The additional element is only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claims 8-12, and 14:
Claims 8-12, and 14 incorporates substantively all the limitations of Claims 1-5 and 7 in a computer-program product comprising one or more non-transitory machine-readable storage media, including stored instructions configured to cause a computing system to perform a set of actions including (these claim limitations appear to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f)) and does not appear to integrate the abstract idea into a particular application; thus, the claim is subject matter ineligible as it does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements, alone or in combination, and do not provide significantly more than the abstract idea itself); thus, Claims 8-12, and 14 are rejected for reasons set forth in the rejection of Claims 1-5 and 7, respectively.
Regarding Claims 15-19:
Claims 15-19 incorporates substantively all the limitations of Claims 1-5 in a system comprising:
one or more processors; one or more non-transitory computer-readable media storing instructions, which, when executed by the system, cause the system to perform a set of actions including: (these claim limitations appear to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f)) and does not appear to integrate the abstract idea into a particular application; thus, the claim is subject matter ineligible as it does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements, alone or in combination, and do not provide significantly more than the abstract idea itself); thus, Claims 15-19 are rejected for reasons set forth in the rejection of Claims 1-5, respectively.
Regarding Claims 6, 13, and 20:
Claims 6, 13, and 20 are subject-matter eligible and not rejected under 35 U.S.C. 101 as the limitations recite ... based at least in part on determining that the second trained machine learning model does not satisfy the one or more conditions, adding the second labeled feedback to at least the first set of data items without initiating retraining of the second trained machine learning model; where these limitations note the second learning model and the conditions for not initiating retraining. The dependent Claims in combination with their respective Independent Claims provide a reasonable basis for integration into a practical application under Subject Matter Eligibility Analysis Step 2A Prong 2.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 7-9, 14, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Callot et al., US-12033048-B1, in view of Nguyen et al., “Active Learning Using Pre-clustering”, in view of Yi et al., US-20230012784-A1.
Regarding Claim 1:
A computer-implemented method comprising:
(Callot, Fig. 9. Figure 9 shows an example computer system (w/ processor(s), interfaces, system memory, etc.) to implement the operations for anomaly detection services; thus, a computer-implemented method).
determining a first accuracy score of a trained machine learning model at determining anomalies in a first set of data items at least in part by:
(Callot, Fig. 4. Figure 4 shows the process/method of training an anomaly detection system to process received data to score (408) and determining potential anomalies based on threshold for the first set of data items; thus, the process determines a first accuracy score of a trained machine learning model based on the first set of data items that are processed).
providing a first unlabeled version of the first set of data items to the trained machine learning model as a first set of inputs to generate a first set of outputs of the trained machine learning model, wherein the first set of outputs is labeled with a first set of anomaly scores; and
(Callot, Fig. 2; Fig. 4; Columns 5-6: 66-14, “At 404, the anomaly detection system is trained and deployed using the training data ... the training data is historical observations of the metrics to monitor … As such … the training data is unlabeled … the one or more models are deployed, data is received for anomaly detection at 406. The trained anomaly detection system is used to process the received data to score the data and determine if there is a potential anomaly based on one or more thresholds at 408. … one or more of the models 202 are used to generate a score and the anomaly decider 214 determines if that score indicates that there is an anomaly based one or more thresholds”; Column 2, Lines 28-31, “Each anomaly scorer associates to each new data point a score based on the statistical rarity of the observation (e.g., with respect to a training set or existing data)”. The anomaly detection system is trained and shown via Figure 4’s process/method. The unlabeled training data is used to train the trained machine learning model. The model then receives data (system is provided the data; Fig. 2: DATA) for anomaly detection (406); which is interpreted by the examiner as providing a first unlabeled version of the first set of data items to the trained machine learning model as the received set of data has not be labeled with an anomaly score (as shown within Fig. 2). The Anomaly Detection System generates labels via the Anomaly Scorer (Fig. 2:202) for the input data (which is the provided unlabeled version of the first set of data); thus, interpreted by the examiner as generating a first set of outputs of the trained machine learning model, wherein the first set of outputs is labeled with a first set of anomaly scores)).
comparing the first set of outputs of the trained machine learning model to a first labeled version of the first set of data items to determine a first set of incorrectly labeled outputs;
(Callot, Fig. 2; Column 7, Lines 4-6, “A feedback set includes the score associated with the feedback and the time of the datapoint that was detected as potentially anomalous”. Figure 2 shows the Anomaly Decider (214) which takes the ANOMALY SCORES/FEATURES and compares the first set of outputs with thresholds (which represents a first labeled version of data of the first set of data items as they are based off the first set of data items which were used to train) to determine the first anomalies (incorrectly labeled) within the dataset; thus, the positive feedback sets containing positive feedback for being anomalous is interpreted as a first set of incorrectly labeled outputs when compared to a first labeled version of data).
determining a contiguous anomaly score value region that includes:
(Callot, Fig. 3; Fig. 4; Column 5, Lines 30-32, “This spike is given, by one or more anomaly detection models, a severity score 306 indicating its likelihood of being anomalous …”. Figure 3 shows the UI for processing new data which assigns a severity score (306: the severity score indicating the likelihood of being anomalous) within a region (308) of the contiguous anomaly scores (300: the graph indicates that the anomalous scores are within a continuous numerical scale). Thus, the system determines a contiguous anomaly score value region that includes thresholds (Fig. 4: 408)).
a threshold portion of the first set of incorrectly labeled outputs,
(Callot, Fig. 2; Fig. 3: 308; Fig. 5; Column 1, Lines 50-53, “… feedback is used ... to modify the threshold used to classify observations as anomalous …”; Column 5, Lines 20-22, “... the feedback incorporator 212 tweaks 20 the anomaly decider 214 to modify the threshold used to classify observations to output as anomalous …”. Figure 2 denotes an Anomaly Decider (214) which makes decisions on data points that would be considered anomalous (incorrectly labeled) based on a threshold. Thus, the Anomaly Detection System (200) comprises a decider (214) which determines an anomaly score value region (like Figure 3’s graph’s spike (308) indicating a potential anomaly region) that contains a portion of incorrectly labeled outputs based on a threshold portion).
one or more outputs of the first set of outputs labeled as anomalous, and one or more outputs of the first set of outputs labeled as not anomalous;
(Callot, Fig. 2; Fig. 3; Fig. 4: 408; Column 9, Lines 25-40, “
PNG
media_image1.png
103
508
media_image1.png
Greyscale
”. Figure 4: 408 denotes the step of utilizing the Anomaly Detection System to process the received data to be able to score and determine potential anomalies based on thresholds; where the feedback sets are generated for positive/negative feedback (anomalous/non-anomalous feedback, respectively); thus, interpreted as one or more outputs of the first set of outputs being labeled as anomalous or not anomalous based on being larger or smaller than the threshold τ and place within respective feedback sets).
receiving a second set of data items that have not been labeled;
(Callot, Fig. 2: DATA; Column 9, Lines 52-54, “… a ranking is produced by the base anomaly detection system scores. This is repeated for every new coming batch”. Figure 2 shows the process of receiving sets of data items (which is done for every new batch including a second set of data items) that have not been labeled (as the detection system has not assigned an anomaly score (label))).
providing the second set of data items to the trained machine learning model as a second set of inputs to generate a second set of outputs of the trained machine learning model, wherein the second set of outputs is labeled with a second set of anomaly scores;
(Callot, Fig. 2; Fig. 4; Columns 5-6: 66-14, “At 404, the anomaly detection system is trained and deployed using the training data ... the training data is historical observations of the metrics to monitor … As such … the training data is unlabeled … the one or more models are deployed, data is received for anomaly detection at 406. The trained anomaly detection system is used to process the received data to score the data and determine if there is a potential anomaly based on one or more thresholds at 408. … one or more of the models 202 are used to generate a score and the anomaly decider 214 determines if that score indicates that there is an anomaly based one or more thresholds”; Column 2, Lines 28-31, “Each anomaly scorer associates to each new data point a score based on the statistical rarity of the observation (e.g., with respect to a training set or existing data)”. The anomaly detection system is trained and shown via Figure 4’s process/method. The model then receives data (system is provided the data; Fig. 2: DATA) for anomaly detection (406); which is interpreted by the examiner as providing the second set of data items to the trained machine learning model as the received set of data has not be labeled with an anomaly score (as shown within Fig. 2). The Anomaly Detection System generates labels via the Anomaly Scorer (Fig. 2:202) for the input data; thus, interpreted by the examiner as generating a second set of outputs of the trained machine learning model, wherein the second set of outputs is labeled with a second set of anomaly scores).
determining an updated accuracy score of the trained machine learning model at determining anomalies in a superset of data items comprising the first set of data items and the second set of data items at least in part by:
(Callot, Fig. 4: 412; Fig. 5: 412. Figures 4 and 5 show 412 which adjusts thresholds used to determine potential anomalies (which is adjusting the sensitivity). This is interpreted as determining an updated accuracy score of the trained machine learning model as the model incorporates utilizing threshold to determine anomalies and scoring them with anomaly scores (Fig. 4:408). Severity/anomaly scores are now updated due to adjusting the sensitivity and then the next batch of received data can be processed (414 -> 406); thus, the training machine learning model is now takes into consideration the superset of data items comprising all batches of received data (including the first set and second set))).
selecting a second subset of data items within the contiguous anomaly score value region, wherein the second subset of data items has fewer items than the second set of data items;
(Callot, Fig. 3: 308; Column 5, Lines: 30-35, “This spike is given, by one or more anomaly detection models, a severity score 306 indicating its likelihood of being anomalous ... the spike is highlighted by potential anomaly box 308”. Fig. 3: 308 denotes a POTENTIAL ANOMAL box (which interpreted as a score value region by the examiner as the box indicates a region of anomaly scores within a region). Thus, the anomaly detections models are selecting a second subset of data items within the contiguous anomaly score value region that indicate potential anomalies within the totality of the graph. The second subset of data items has fewer items than the second set of data items as shown in the graph (Fig. 3: 308 < Fig 3: Total ANOMALY RESULT 300)).
clustering the second subset of data items into a plurality of clusters based at least in part on one or more feature values of the second subset of data items;
(Callot, Fig. 3; Fig. 6; Column 9, Lines 25-40, “
PNG
media_image1.png
103
508
media_image1.png
Greyscale
”. Figure 6 shows the process/method for adjusting what is considered an anomaly via clustering. Fig. 6: 604 selects candidates via the anomaly detection system to classify the scores that are larger than a threshold; thus, the second subset of data items (Fig. 3: 308 POTENTIAL ANOMALY box) is clustered based on selected candidates (scores larger than threshold) via clustering the vectors (Fig. 6: 604 & 610). Thus, the clustering is based on feature values (severity/anomaly scores)).
...
collecting labeled feedback for the ... ;
(Callot, Fig. 5: 502. Figure 5:502 notes initializing one or more feedback sets; thus, Callot’s methodology contains collecting labeled feedback (positive=anomalous or negative=non-anomalous) for the first and second subsets of data items).
determining a second accuracy score at least in part by comparing, from the trained machine learning model, ...
(Callot, Fig. 3 & 4; Column 5, Lines 30-32, “This spike is given, by one or more anomaly detection models, a severity score 306 indicating its likelihood of being anomalous …”. Figure 4 shows the process/method of training an anomaly detection system to process received data to score (408) and determining potential anomalies and scoring based on threshold. Thus, Fig. 3: 306 shows the severity/anomaly score (interpreted as an accuracy score) of the POTENTIAL ANOMALY (308) (a second accuracy score) which was compared to threshold and labeled outputs which is shown in Figure 3).
...
based at least in part on the updated accuracy score, determining whether the trained machine learning model satisfies one or more conditions for retraining the trained machine learning model; based at least in part on determining that the trained machine learning model satisfies the one or more conditions, initiating retraining of the trained machine learning model.
(Callot, Fig. 4&5. Figure 4 shows training via step 404, receiving data via step 406, scoring via step 408, and then steps 409-410 requests feedback to update sensitivity in step 412 and adjust/learn relevancy via step 414. This adjusting/learning step of 414 is then used to update the model’s sensitivity for determining anomalies; thus, relearning/retraining as the model is now receiving data and the model has been updated (where the one or more conditions for retraining are shown within Fig. 5: 412)).
However, Callot does not explicitly teach:
selecting a third subset of data items from the second subset of data items such that:
(Nguyen, Abstract, “”The algorithm first constructs a classifier on the set of the cluster representatives ... The proposed model allows to select the most representative samples ...”; Page 5, Column 2, Section 3.3: Paragraph 1, “The selection criterion gives priority to ... samples which are cluster representatives ... within the set of cluster representatives, one should start with the highest density clusters first”. Nguyen teaches selecting a subset of data item from each cluster which is noted as cluster representatives into a set of cluster representatives; thus, the set of cluster representative is interpreted as the third subset of data items from the plurality of clusters (which is interpreted as the second subset of data items as each clusters is a subset of the initial data)).
the third subset has fewer items than the second subset, and
(Nguyen, Abstract, “The algorithm first constructs a classifier on the set of the cluster representatives ... The proposed model allows to select the most representative samples ...”; Page 5, Column 2, Section 3.3: Paragraph 1, “The selection criterion gives priority to ... samples which are cluster representatives ... within the set of cluster representatives, one should start with the highest density clusters first”. Nguyen teaches selecting a third subset of data items from each cluster which is noted as cluster representatives into a set of cluster representatives; thus, the set of cluster representatives is fewer in items than the second subset as there is only a one or a few representatives of each cluster).
the third subset has one or more data items in each cluster of the plurality of clusters;
(Nguyen, Abstract, “The algorithm first constructs a classifier on the set of the cluster representatives ... The proposed model allows to select the most representative samples ...”; Page 5, Column 2, Section 3.3: Paragraph 1, “The selection criterion gives priority to ... samples which are cluster representatives ... within the set of cluster representatives, one should start with the highest density clusters first”. The third subset has one or more data items in each cluster as the set is made up of at least one representative from each cluster)).
... third subset of data items;
(Nguyen, Page 5, Column 2, Section 3.3: Paragraph 1, “The selection criterion gives priority to ... samples which are cluster representatives ... within the set of cluster representatives, one should start with the highest density clusters first”. The set of cluster representatives, as noted above, is the third subset of data items).
... a third subset of labeled outputs of the third subset of data items ...; and
(Nguyen, Page 7, Column 1, Section 5: Paragraph 1, “The model allows to select most representative training examples as well as to avoid repeatedly labeling samples in same cluster, leading to better performance than the current methods”; Page 4, Figure 3. The third subset of labeled output are taught by labeling the third subset of data items which are the most representative and noted as cluster representative (and noted within the learning algorithm of Figure 3)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the method of Callot which comprises an anomaly detection system for determinations with Nguyen’s explicit teaching of selecting of another subset where there is one or more data items from each of plurality of clusters to represent clusters within a subset as this leads to incorporation of clustering into active learning, avoid repeated labeling, stabilized training sets, and using cluster information (Nguyen, Page 7, Column 1, Section 5: Paragraph 1, “The paper has proposed a formal model for incorporation of clustering into active learning. The model allows to select most representative training examples as well as to avoid repeatedly labeling samples in same cluster, leading to better performance than the current methods. ... In addition to closeness to the classification boundary, the selection criterion gives priority also to the representatives of the dense clusters, making the training set statistically stable. The method was restricted to linear logistic regression as the main purpose of the paper is to show the advantage of using clustering information. We have succeeded in that goals for the given datasets”).
However, Callot/Nguyen do not explicitly teach:
combining the first accuracy score and the second accuracy score;
Nevertheless, Yi teaches:
combining the ... accuracy score and the ... accuracy score;
(Yi, Page 7, Column 1, [0109], “Reference is made back to FIG. 8B, the line item detector 520 calculates the overall accuracy level for the candidate group by aggregating the cluster accuracy levels determined for the certain number of candidate clusters (block 850) ...”. Yi teaches aggregating (interpreted as combining as they are being added together) the accuracy levels (interpreted as accuracy scores) for the multiple candidate clusters).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the method of Callot/Nguyen’s anomaly detection system with Yi’s explicit combination of accuracy score to calculate the overall accuracy of candidate groups, analysis, selection and review accuracy trends (Yi, Page 8, Column 1, [0110-0113] “... where the line item detector 520 selects the target group from the plurality of candidate groups based on the overall accuracy levels determined for the plurality of candidate groups. Generally, the accuracy trend for the candidate groups may vary as the numbers of candidate clusters are comprised in the candidate groups. ... FIG. 9D depicts a curve 910 of overall accuracy levels calculated for different numbers of candidate clusters in accordance with some embodiments of the present disclosure. As can be seen, the overall accuracy level increased from one cluster and reaches the highest value 912 when the number of candidate clusters reaches 6. ... a candidate group with a highest overall accuracy level may be selected as the target group for classifying ... the overall accuracy levels for the candidate groups may be sorted to find the highest overall accuracy level ... if the overall accuracy level converges to a relatively stable level as the number of candidate clusters increases, the lowest number may be selected”).
Regarding Claim 2:
Callot/Nguyen/Yi teach the method of Claim 1 and Callot further teaches:
sending a notification ..., wherein the notification provides a summary comprising the updated accuracy score and a time for the retraining.
(Callot, Fig. 3-5. Figure 3 teaches sending a notification by highlight the potential anomaly (308) and requesting binary feedback (304) which can trigger the retraining/relearning via adjusting the thresholds of the model (Fig. 4 & 5). Thus, with the retraining and using the new datasets/findings the accuracy score will be updated as the threshold is adjusted which directly affects the severity/anomaly score).
... to an administrator of the trained machine learning model
(Yi, Page 3, Column 2, [0054], “User portal 83 provides access to the cloud computing environment for consumers and system administrators”. Yi teaches a portal for sending notifications and allow users/administrators of the trained machine learning model).
The motivation of Claim 1’s combination is maintained.
Regarding Claim 7:
Callot/Nguyen/Yi teach the method of Claim 1 and Nguyen further teaches:
... tuning one or more hyperparameters of the trained machine learning model based at least in part on the third subset of labeled outputs.
(Nguyen, Page 4, Column 2, Section 3.2, Paragraph 1, “Fixing the cluster representatives ck, the likelihood depends only on the parameters a and b ... Starting with an initial guess a0 and b0 the parameters a and b are updated iteratively ...”. Nguyen teaches selecting representative samples, obtaining the labels, estimating the class label model from the representative samples, and tuning the hyperparameters iteratively to update the model based on the third subset of labeled outputs (labeled cluster representatives)).
Regarding Claims 8-9 and 14:
Claims 8-9 and 14 incorporates substantively all the limitations of Claims 1-2 and 7 in a computer-program product comprising one or more non-transitory machine-readable storage media, including stored instructions configured to cause a computing system to perform a set of actions including (Callot, Column 1, Lines 47-50, “The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for anomaly detection and changes to that detection using feedback”); thus, Claims 8-9 and 14 are rejected for reasons set forth in the rejection of Claims 1-2 and 7, respectively.
Regarding Claims 15-16:
Claims 15-16 incorporates substantively all the limitations of Claims 1-2 in a system comprising:
one or more processors; one or more non-transitory computer-readable media storing instructions, which, when executed by the system, cause the system to perform a set of actions including: (Callot, Column 1, Lines 47-50, “The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for anomaly detection and changes to that detection using feedback”; Fig. 9. Fig. 9 shows the processors, system memory (contain code; thus, instructions) to perform the anomaly detection service); thus, Claims 15-16 are rejected for reasons set forth in the rejection of Claims 1-2, respectively.
Examiner Comments
Claims 3-5, 10-12, and 17-19 are currently rejected under 35 USC § 101 only. Claims 6, 13, and 20 are currently only objected to. A complete and thorough search was performed for these claims; however no prior art was uncovered that teach or fairly suggest the features recited claims. Specifically, none of the prior art of record, either alone or in combination, fairly discloses the limitations of the independent Claim. In particular, the limitations in:
Regarding Claims 3, 10, and 17:
... scheduling the retraining based at least in part on two or more different frequencies by which data items are provided to the trained machine learning model over two or more windows of time, and wherein the retraining is scheduled for a particular window of time of the two or more windows of time.
Regarding Claims 4, 11, and 18:
... wherein the sensors are separately identified and tracked in an anomaly detection platform, and wherein the sensors stream the second subset of data items into the anomaly detection platform using connections that provide sensor-identifying information.
Regarding Claims 5, 12, and 19:
wherein selecting the third subset of data items comprises: randomly selecting a unique data item from the second subset of data items; assigning the unique data item to a particular cluster of the plurality of clusters; and re-performing said randomly selecting if adding the unique data item to the third subset of data items would result in an over-representation of the particular cluster.
Regarding Claims 6, 13, and 20:
... determining a second updated accuracy score of the second trained machine learning model at determining anomalies in a second superset of data items comprising the first set of data items and the third set of data items at least in part by: selecting a fourth subset of data items within a second contiguous anomaly score value region, wherein the fourth subset of data items has fewer items than the third set of data items; clustering the fourth subset of data items into a second plurality of clusters based at least in part on one or more feature values of the fourth subset of data items; selecting a fifth subset of data items from the fourth subset of data items such that: the fifth subset has fewer items than the fourth subset, and the fifth subset has one or more data items in each cluster of the second plurality of clusters; ...
The closest prior art of record is Callot et al., US-12033048-B1, in view of Nguyen et al., “Active Learning Using Pre-clustering”, in view of Yi et al., US-20230012784-A1; where the computer implemented methodology of Callot/Nguyen/Yi teaches an anomaly detection system that determines accuracy scores, handles multiple labeled and unlabeled datasets, compares outputs to determine anomalous data, utilizes clustering, and initiates retraining. However, Callot/Nguyen/Yi do not explicitly disclose retraining based on frequencies of data items, scheduling retraining for a particular window, sensors separately identifying/tracking in an anomaly detection platform, sensor streams to provide sensor-identifying information, random selection to and re-performing of random selection when a cluster is overrepresented, a second contiguous value region, and utilizing a 4th-5th subset for comparison of labeled feedback. Thus, the combination of these three prior arts do not disclose the expressions defined in limitations above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IBRAHIM RAHMAN whose telephone number is (703)756-1646. The examiner can normally be reached M-F 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.R./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights