DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of prior-filed a U.S. National Stage Application filed under 35 U.S.C. §371, International Patent Application No. PCT/JP2020/035622, filed on 18 September 2020,, which is acknowledged.
Drawings
The drawings were received on 03/14/2023. These drawings are acceptable.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on the following date(s): 07/23/2024 and 03/14/2023 have been considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without significantly more.
Claim 1: Does claim fall within a statutory category? Yes.
Step 2A Prong 1: Evaluate whether the claim recites a judicial exception.
… determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, information regarding over-detection data over-detected (Considered directed to a Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) as claimed; see MPEP § 2106.04(a)(2), subsection III)
Step 2A Prong 2: Evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception
The preamble is deemed insufficient to transform the judicial exception to a patentable invention because the preamble generally links the use of a judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h).
processing circuitry configured to: determine … an abnormality detection system that uses the first model, and …. a second model generated based on the over-detection data; (Deemed insufficient to transform the judicial exception to a patentable invention because the recitation merely include instructions to implement an abstract idea on a computer, or merely use a computer as a tool to perform an abstract idea; Thus claim limitations amount to mere instructions to apply the judicial exception using a computer/computing environment as a tool, as discussed in MPEP § 2106.05(f).)
(Deemed insufficient to transform the judicial exception to a patentable invention because the recitation generally links the use of a judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h), e.g. Type of acquired learning data as claimed.)
and notify of a result of a determination. (Deemed insufficient to transform the judicial exception to a patentable invention because the recitation is directed to insignificant solution activity, e.g. receiving or transmitting data over a network)
The additional elements do not appear to be sufficient to transform the judicial exception into a practical application at Step 2A as analyzed above.
Step 2B: Evaluates whether the claim as a whole/in combination integrates the recited judicial exception into a practical application of the exception
The claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception and fail to integrate the abstract into practical application.
The additional limitations are directed to elements that generally link the use of a judicial exception to a particular technological environment or field of use and merely invoke the use of computer technology as a tool for applying the judicial exception; and that generally link the use of a judicial exception to a particular technological environment and directed to invoking computers or other machinery merely as a tool to perform the claimed process/judicial exception.
Secondly, the limitations directed to insufficient to transform the judicial exception to a patentable invention because the recitation is directed to insignificant solution activity for as noted above.
The courts have deemed these types of activity as well-known routine and convectional, see evidences noted below:
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink." (emphasis added));
These types of claimed elements cannot transform the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible.
Claim 2: Does claim fall within a statutory category? Yes.
Step 2A Prong 1: Evaluate whether the claim recites a judicial exception.
(Considered directed to a Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) as claimed; see MPEP § 2106.04(a)(2), subsection III; And alternatively: Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations (see MPEP § 2106.04(a)(2), subsection I))
Step 2A Prong 2: Evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception
The preamble is deemed insufficient to transform the judicial exception to a patentable invention because the preamble generally links the use of a judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h).
wherein the processing circuitry is further configured to … (Deemed insufficient to transform the judicial exception to a patentable invention because the recitation merely include instructions to implement an abstract idea on a computer, or merely use a computer as a tool to perform an abstract idea; Thus claim limitations amount to mere instructions to apply the judicial exception using a computer/computing environment as a tool, as discussed in MPEP § 2106.05(f).)
The additional elements do not appear to be sufficient to transform the judicial exception into a practical application at Step 2A as analyzed above.
Step 2B: Evaluates whether the claim as a whole/in combination integrates the recited judicial exception into a practical application of the exception
The claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception and fail to integrate the abstract into practical application.
The additional limitations are directed to elements that generally link the use of a judicial exception to a particular technological environment or field of use and merely invoke the use of computer technology as a tool for applying the judicial exception; and that generally link the use of a judicial exception to a particular technological environment and directed to invoking computers or other machinery merely as a tool to perform the claimed process/judicial exception.
These types of claimed elements cannot transform the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible.
Claim 3: Does claim fall within a statutory category? Yes.
Step 2A Prong 1: Evaluate whether the claim recites a judicial exception.
(Considered directed to a Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) as claimed; see MPEP § 2106.04(a)(2), subsection III; And alternatively: Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception
The preamble is deemed insufficient to transform the judicial exception to a patentable invention because the preamble generally links the use of a judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h).
wherein the processing circuitry is further configured to … (Deemed insufficient to transform the judicial exception to a patentable invention because the recitation merely include instructions to implement an abstract idea on a computer, or merely use a computer as a tool to perform an abstract idea; Thus claim limitations amount to mere instructions to apply the judicial exception using a computer/computing environment as a tool, as discussed in MPEP § 2106.05(f).)
The additional elements do not appear to be sufficient to transform the judicial exception into a practical application at Step 2A as analyzed above.
Step 2B: Evaluates whether the claim as a whole/in combination integrates the recited judicial exception into a practical application of the exception
The claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception and fail to integrate the abstract into practical application.
The additional limitations are directed to elements that generally link the use of a judicial exception to a particular technological environment or field of use and merely invoke the use of computer technology as a tool for applying the judicial exception; and that generally link the use of a judicial exception to a particular technological environment and directed to invoking computers or other machinery merely as a tool to perform the claimed process/judicial exception.
These types of claimed elements cannot transform the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible.
Claim 4: Does claim fall within a statutory category? Yes.
Step 2A Prong 1: Evaluate whether the claim recites a judicial exception.
. (Considered directed to a Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) as claimed; see MPEP § 2106.04(a)(2), subsection III; And alternatively: Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception
The preamble is deemed insufficient to transform the judicial exception to a patentable invention because the preamble generally links the use of a judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h).
wherein the processing circuitry is further configured to … (Deemed insufficient to transform the judicial exception to a patentable invention because the recitation merely include instructions to implement an abstract idea on a computer, or merely use a computer as a tool to perform an abstract idea; Thus claim limitations amount to mere instructions to apply the judicial exception using a computer/computing environment as a tool, as discussed in MPEP § 2106.05(f).)
The additional elements do not appear to be sufficient to transform the judicial exception into a practical application at Step 2A as analyzed above.
Step 2B: Evaluates whether the claim as a whole/in combination integrates the recited judicial exception into a practical application of the exception
The claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception and fail to integrate the abstract into practical application.
The additional limitations are directed to elements that generally link the use of a judicial exception to a particular technological environment or field of use and merely invoke the use of computer technology as a tool for applying the judicial exception; and that generally link the use of a judicial exception to a particular technological environment and directed to invoking computers or other machinery merely as a tool to perform the claimed process/judicial exception.
These types of claimed elements cannot transform the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible.
Claim 5: Does claim fall within a statutory category? Yes.
Step 2A Prong 1: Evaluate whether the claim recites a judicial exception.
predetermined value. (Considered directed to a Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) as claimed; see MPEP § 2106.04(a)(2), subsection III; And alternatively: Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception
The preamble is deemed insufficient to transform the judicial exception to a patentable invention because the preamble generally links the use of a judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h).
wherein the processing circuitry is further configured to … (Deemed insufficient to transform the judicial exception to a patentable invention because the recitation merely include instructions to implement an abstract idea on a computer, or merely use a computer as a tool to perform an abstract idea; Thus claim limitations amount to mere instructions to apply the judicial exception using a computer/computing environment as a tool, as discussed in MPEP § 2106.05(f).)
The additional elements do not appear to be sufficient to transform the judicial exception into a practical application at Step 2A as analyzed above.
Step 2B: Evaluates whether the claim as a whole/in combination integrates the recited judicial exception into a practical application of the exception
The claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception and fail to integrate the abstract into practical application.
The additional limitations are directed to elements that generally link the use of a judicial exception to a particular technological environment or field of use and merely invoke the use of computer technology as a tool for applying the judicial exception; and that generally link the use of a judicial exception to a particular technological environment and directed to invoking computers or other machinery merely as a tool to perform the claimed process/judicial exception.
These types of claimed elements cannot transform the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible.
Claim 6: Does claim fall within a statutory category? Yes.
Step 2A Prong 1: Evaluate whether the claim recites a judicial exception.
(Considered directed to a Mental processes – concepts performed in the human mind (including an observation, evaluation, judgment, opinion) as claimed; see MPEP § 2106.04(a)(2), subsection III; And alternatively: Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception
The preamble is deemed insufficient to transform the judicial exception to a patentable invention because the preamble generally links the use of a judicial exception to a particular technological environment or field of use, see MPEP 2106.05(h).
wherein the processing circuitry is further configured to … (Deemed insufficient to transform the judicial exception to a patentable invention because the recitation merely include instructions to implement an abstract idea on a computer, or merely use a computer as a tool to perform an abstract idea; Thus claim limitations amount to mere instructions to apply the judicial exception using a computer/computing environment as a tool, as discussed in MPEP § 2106.05(f).)
The additional elements do not appear to be sufficient to transform the judicial exception into a practical application at Step 2A as analyzed above.
Step 2B: Evaluates whether the claim as a whole/in combination integrates the recited judicial exception into a practical application of the exception
The claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception and fail to integrate the abstract into practical application.
The additional limitations are directed to elements that generally link the use of a judicial exception to a particular technological environment or field of use and merely invoke the use of computer technology as a tool for applying the judicial exception; and that generally link the use of a judicial exception to a particular technological environment and directed to invoking computers or other machinery merely as a tool to perform the claimed process/judicial exception.
These types of claimed elements cannot transform the judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. This claim is not patent eligible.
Regarding claims 7 and 8, the limitations are similar with claim 1 limitations and thus rejected under the same rationale.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-8 are rejected under 35 U.S.C. 102(a)(1) and 35 U.S.C. 102(a)(2) as being anticipated by Hearty et al. (US 20200280578, hereinafter ‘Hearty’).
Regarding independent claim 1, Hearty teaches a determination device comprising: processing circuitry configured to: (in [0010] In addition, it should be understood that embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored on non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and/or application specific integrated circuits (“ASICs”)…)
determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data; (As depicted in Fig. 12; And in [0091] FIG. 12 is a flowchart that illustrates an OAO feature drift hardening process 1200, according to embodiments described herein. FIG. 12 is described with respect to the fraud prevention server 1135 of FIG. 11… [0094] The alerting component 1210 may be a software service that distributes alerts related to feature drift and model evaluation performance to relevant groups including individuals that maintain the models and users of the models. The OAO drift weighting component 1212 may be a software module that enables the setting of manually or automatically derived drift weights [and information regarding a second model generated based on the over-detection data], such as event recency weights and the application of said weights as a feature within models. The OAO model set 1214 may be a collection of subcomponents including: 1) longer-term models 1218, intended to learn longer-term trends from mostly time-series features or alternatively to model drift components using a subset of information such as drift weighting and recency features, and 2) shorter-term models 1220 [alternatively the plurality of models including information regarding a second model generated based on the over-detection data], intended to learn short-term state from mostly stationary features without the impact of long-term drift. [0095] The OAO model retraining component 1216 receives drift monitoring results from the OAO drift monitoring component 1208 [information regarding over-detection data over-detected by an abnormality detection system that uses the first model] and model monitoring results from the OAO model evaluation and monitoring component 1226 [determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, information regarding over-detection data over-detected by an abnormality detection system that uses the first model]. The OAO model retraining component 1216 [determine necessity of relearning of a first model] outputs one or more retrained models to the OAO model set 1214 in response to determining that retraining is necessary based on at least one of the drift monitoring results or the model monitoring results.)
and notify of a result of a determination. (in [0094] The alerting component 1210 may be a software service that distributes alerts [notify of a result of a determination] related to feature drift and model evaluation performance to relevant groups including individuals that maintain the models and users of the models…; And in [0097] The OAO model evaluation and monitoring component 1226 may be a software component designed to assess trained and live model performance through statistical evaluation, compare the results of evaluation to defined performance requirements, and trigger an alert [notify of a result of a determination] when the comparison determines that the performance deviates from requirement thresholds. Lastly, the OAO model output visualization component 1228 may be designed to support analysis and model diagnostic activity by individuals that maintain the models or users of the models.)
Regarding claim 2, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value. (in [0093] The feature calculator 1206 may be a software module that is used to generate “features” (data input variables) for modeling. The OAO drift monitoring component 1208 may be a software module that measures the degree of feature drift [a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data] across individual features and combinations of 2 . . . n features (where n is the total number of features in the set, for example, some or all of the behavioral features described above). These combinations may be problem specific or automatically defined (e.g., every combination of features may be assessed). The OAO drift monitoring component 1208 may also compare drift against a defined threshold and trigger an alert and retraining activity [wherein the processing circuitry is further configured to determine that the relearning of the first model…] when the comparison determines that the drift exceeds a certain threshold [wherein the processing circuitry is further configured to determine that the relearning of the first model is necessary when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value].)
Regarding claim 3, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary in a case where a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value. (in [0105] The fraud prevention server 1135 executes the OAO model retraining component 1216 to consume information generated by the OAO drift and model monitoring components 1208 and 1226, and identify when a retraining of the model is necessary. This decision is based on statistical analysis [determine that the relearning of the first model is necessary in a case where a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard…] of the output of the OAO model evaluation and monitoring component 1226. Model retraining [determine that the relearning of the first model is necessary…] may be initiated when either model score distribution (e.g. score central tendency, proportion of traffic identified as high risk) begins to deviate beyond accepted levels [determine that the relearning of the first model is necessary in a case where a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value], when the fraud prevention server 1135 executes the OAO drift monitoring component 1208 and identifies a consistently higher degree of drift, or when a defined period of time has passed (e.g., one week, one month, three months, or some other temporal period). In addition, model retraining may also be manually initiated.)
Regarding claim 4, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a loss function of the second model exceeds a predetermined value. ([0093] The feature calculator 1206 may be a software module that is used to generate “features” (data input variables) for modeling. The OAO drift monitoring component 1208 may be a software module that measures the degree of feature drift [determine that the relearning of the first model is necessary when a loss function…] across individual features and combinations of 2 . . . n features (where n is the total number of features in the set, for example, some or all of the behavioral features described above). These combinations may be problem specific or automatically defined (e.g., every combination of features may be assessed). The OAO drift monitoring component 1208 may also compare drift against a defined threshold and trigger an alert and retraining activity when the comparison determines that the drift exceeds a certain threshold [determine that the relearning of the first model is necessary when a loss function of the second model exceeds a predetermined value].)
Regarding claim 5, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value. (in [0105] The fraud prevention server 1135 executes the OAO model retraining component 1216 to consume information generated by the OAO drift and model monitoring components 1208 and 1226, and identify when a retraining of the model is necessary. This decision is based on statistical analysis [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value.] of the output of the OAO model evaluation and monitoring component 1226. Model retraining [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system…] may be initiated when either model score distribution (e.g. score central tendency, proportion of traffic identified as high risk) begins to deviate beyond accepted levels [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value], when the fraud prevention server 1135 executes the OAO drift monitoring component 1208 and identifies a consistently higher degree of drift, or when a defined period of time has passed (e.g., one week, one month, three months, or some other temporal period). In addition, model retraining may also be manually initiated.)
Regarding claim 6, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value. (in [0096] The OAO model selector 1222 may be a mathematical function designed to select which OAO models to execute against a sample based on the observed drift of the sample [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value]. The score resolution component 1224 may be a software component designed to combine the scores from various OAO models into a single result. The combination of the scores from various OAO models into the single result may be achieved with a problem-specific regression function, or using ensemble resolution techniques such as stacking or bucketing. [0097] The OAO model evaluation and monitoring component 1226 [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value] may be a software component designed to assess trained and live model performance through statistical evaluation, compare the results of evaluation to defined performance requirements, and trigger an alert when the comparison determines that the performance deviates from requirement thresholds [a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value]… [0105] The fraud prevention server 1135 executes the OAO model retraining component 1216 to consume information generated by the OAO drift and model monitoring components 1208 and 1226 [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value], and identify when a retraining of the model is necessary. This decision is based on statistical analysis of the output of the OAO model evaluation and monitoring component 1226. Model retraining may be initiated when either model score distribution (e.g. score central tendency, proportion of traffic identified as high risk) begins to deviate beyond accepted levels, when the fraud prevention server 1135 executes the OAO drift monitoring component 1208 and identifies a consistently higher degree of drift, or when a defined period of time has passed (e.g., one week, one month, three months, or some other temporal period)…)
Regarding claims 7 and 8, the limitations are similar to claim 1 limitations and thus rejected under the same rationale.
Claims 1-8 are rejected under 35 U.S.C. 102(a)(1) and 35 U.S.C. 102(a)(2) as being anticipated by Masud et al. (US 20120054184, hereinafter ‘Masud’).
Regarding independent claim 1, Masud teaches a determination device comprising: processing circuitry configured to: (in [0256] Referring to FIG. 23, a block diagram of a computing device 102 is shown in which the illustrative embodiments may be implemented. In particular, the detection of novel data classes, as described in any of the illustrative embodiments, may be implemented on the computing device 602. Computer-usable program code or instructions implementing the processes used in the illustrative embodiments may be located on the computing device 602… [0257] The processor unit 605 serves to execute instructions for software that may be loaded into the memory 607. The processor unit 605 may be a set of one or more processors or may be a multi-processor core, …)
determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data; (As depicted in Fig. 1; And in [0051] Classification models 106 comprise the data classifiers in data classification manager 100 and may be used to classify data stream 104. Classification models 106 may comprise a single model or an ensemble of models [determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, …]. In one embodiment, classification models 106 comprise an ensemble of N models, and each model may be trained to classify data instances using a labeled, data chunk […a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal]. The ensemble may also be continuously updated so that it represents the most recent concept in the stream. For example, the update may be performed in one embodiment as follows: when a new classification model is trained, one of the existing models in the ensemble may be replaced by the new model, if necessary [information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data]. The victim model may be chosen by evaluating the error rate of each of the existing models in the ensemble on the latest-labeled chunk [information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data], and discarding the one with the highest error rate.; Alternatively in [0052] In addition, each classification model 106 in the ensemble may detect novel classes within data stream 104. The novel data detection features may be applied to synthetic and real-world data and enable classification models 106 to automatically detect new classes arriving in data stream 104 [information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data], without requiring manual intervention. For example, each classification model 106 processing a data stream 104 may attempt to classify a data instance in the stream. A class may be defined as a novel class if none of the classification models 106 has been trained with that class. Otherwise, if one or more of the classification models 106 has been trained with that class, then that class may be considered an existing class. Data points belonging to the same class should be closer to each other (cohesion) than other data points, and should be far apart from the data points belonging to other classes (separation). [0053] The detection and determination of a novel class [information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data] may comprise the following main aspects. First, a decision boundary may be built during training of the models. Second, the test points falling outside the decision boundary may be declared as filtered outliers, or F-outliers. F-outliers have the potential to be declared novel class instances. Third, the F-outliers may be analyzed to see if there is enough cohesion among themselves (i.e., among the F-outliers) and separation from the training instances. Fourth, where the cohesion and separation is sufficient, the F-outliers may be identified, as instances in a novel class… [0201] Training and update of the models [determine necessity of relearning of a first mode] are shown at line 14. As a model is trained from the training data, a decision boundary around, the training data may be built in order to detect novel classes [determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data;]. Each model also saves the set of features with which it is trained. The newly trained model may replace an existing model in the ensemble…)
and notify of a result of a determination. (As depicted in Fig. 16A-B; And in [0185] The following performance metrics are used for evaluation: M.sub.new=% of novel class instances Misclassified as existing class, F.sub.new=% of existing class instances Falsely identified as novel class, ERR=Total, misclassification error (%) (including M.sub.new and F.sub.new). The initial models are built in each method with the first InitNumber chunks. From the InitNumber+1st chunk onward, first the performances of each, method are evaluated on that chunk, then the chunk is used to update the existing models. InitNumber=3 is used for all experiments. The performance metrics for each chunk are saved [notify of a result of a determination] and aggregated for producing the summary result [notify of a result of a determination].)
Regarding claim 2, the rejection of claim 1 is incorporated and Masud further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0256] - [0257])
determine that the relearning of the first model is necessary when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value. (in [0006] …The method also includes generating, for each of the classification models in the ensemble, a respective decision boundary, and identifying, for each of the classification models in the ensemble, a respective set of filtered outliers that are outside of the respective decision boundary of the classification model. The method also includes determining a cohesion and a separation for the respective set of filtered outliers for each of the classification models in the ensemble. The method also includes determining, by each of the classification models in the ensemble, whether a novel class is detected by the classification model using the cohesion and the separation for the respective set of filtered outliers [determine that the relearning of the first model is necessary when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value], and detecting the novel class in response to a threshold number of the plurality of classification models in the ensemble determining that the novel class is detected [when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value]… [0056] The cohesion and separation analyzer 114 compares the F-outliers to each other and to the existing classes.… [0252] The data classification system may then detect a novel class using the cohesion and the separation of the set of F-outliers, the novel class comprising the set of F-outliers (step 308). For example, in one embodiment, the novel class may be detected [determine that the relearning of the first model is necessary…] using the cohesion and the separation of the set of filtered outliers when a threshold number of the set of F-outliers having the cohesion and the separation that exceeds a predetermined threshold [… when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value]. In another example embodiment, detecting the novel class using the cohesion and the separation of the set of filtered outliers may include detecting the novel class in response to at least a threshold number of the plurality of classification models using the cohesion and the separation for the set of filtered outliers to detect the novel class…)
Regarding claim 3, the rejection of claim 1 is incorporated and Masud further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0256] - [0257])
determine that the relearning of the first model is necessary in a case where a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value. (in [0006] …The method also includes generating, for each of the classification models in the ensemble, a respective decision boundary, and identifying, for each of the classification models in the ensemble, a respective set of filtered outliers that are outside of the respective decision boundary of the classification model. The method also includes determining a cohesion and a separation for the respective set of filtered outliers for each of the classification models in the ensemble. The method also includes determining, by each of the classification models in the ensemble, whether a novel class is detected by the classification model using the cohesion and the separation for the respective set of filtered outliers [determine that the relearning of the first model is necessary in a case where a ratio of the number of types of over- detection data], and detecting the novel class in response to a threshold number [a predetermined standard] of the plurality of classification models in the ensemble determining that the novel class is detected [when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value]… [0056] The cohesion and separation analyzer 114 compares the F-outliers to each other and to the existing classes.… [0252] The data classification system may then detect a novel class using the cohesion and the separation of the set of F-outliers, the novel class comprising the set of F-outliers (step 308). For example, in one embodiment, the novel class may be detected [determine that the relearning of the first model is necessary…] using the cohesion and the separation of the set of filtered outliers when a threshold number of the set of F-outliers having the cohesion and the separation that exceeds a predetermined threshold [when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value]. In another example embodiment, detecting the novel class using the cohesion and the separation of the set of filtered outliers may include detecting the novel class in response to at least a threshold number of the plurality of classification models using the cohesion and the separation for the set of filtered outliers to detect the novel class…)
Regarding claim 4, the rejection of claim 1 is incorporated and Masud further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a loss function of the second model exceeds a predetermined value. ([0051] Classification models 106 comprise the data classifiers in data classification manager 100 and may be used to classify data stream 104. Classification models 106 may comprise a single model or an ensemble of models [determine that the relearning of the first model is necessary when a loss function of the second model exceeds a predetermined value]. In one embodiment, classification models 106 comprise an ensemble of N models, and each model may be trained to classify data instances using a labeled, data chunk. The ensemble may also be continuously updated so that it represents the most recent concept in the stream. For example, the update may be performed in one embodiment as follows: when a new classification model is trained, one of the existing models in the ensemble may be replaced by the new model, if necessary [determine that the relearning of the first model is necessary when a loss function of the second model exceeds a predetermined value]. The victim model may be chosen by evaluating the error rate of each of the existing models [when a loss function of the second model exceeds a predetermined value] in the ensemble on the latest-labeled chunk, and discarding the one with the highest error rate [when a loss function of the second model exceeds a predetermined value].)
Regarding claim 5, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0256] - [0257])
determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value. (in [0148] In one experimental example, let F.sub.n=total novel class instances misclassified as an existing class [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value], F.sub.p=total existing class instances misclassified as a novel class, F.sub.e=total existing class instances misclassified (other than FP), N.sub.c=total novel class instances in the stream, N total instances in the stream. The following performance metrics may be used to evaluate this technique: M.sub.new=% of novel class instances Misclassified as existing class … [0244] Evaluation approach: The following performance metrics may be used for evaluation: M.sub.new=% of novel class instances Misclassified as existing class, F.sub.new=% of existing class instances [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value] Falsely identified as novel class, ERR=Total misclassification error (%) (including M.sub.new and F.sub.new). The initial, models in each method may be built with the first three chunks. From the 4.sup.th chunk onward, first the performances of each method may be evaluated on that chunk, then that chunk may be used to update the existing models [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value]… [0246] Note that although O-F and Lossy-L have lower ERR than DXMiner, they may have higher M.sub.new rates, as they miss most of the novel class instances. This is because both FAE and Lossy-L use the Lossy-L conversion, which, according to Lemma 2, is likely to misclassify more novel class instances as existing class instance (i.e., have higher M.sub.new rates) […when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value]…)
Regarding claim 6, the rejection of claim 1 is incorporated and Masud further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0256] - [0257])
determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value. (in [0253] FIG. 21 is a flowchart that depicts another process for detecting a novel class [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value] in accordance with an illustrative embodiment. For example, in one embodiment, the process described in FIG. 21 may be implemented by the data classification manager 100 in FIG. 1. Upon receiving a data stream comprising a plurality of data points (step 402), the data classification manager may divide the data stream into a plurality of chunks (step 404). The data classification manager may also generate a plurality of classification models to form an ensemble, each of the classification models generated using respective chunks in the plurality of chunks (step 406). The data classification manager may generate, for each of the classification models in the ensemble [a degree of abnormality calculated by the first model], a respective decision boundary (step 408). For each of the classification models in the ensemble, the data classification manager may identify a respective set of filtered outliers that are outside of the respective decision boundary of the classification model (step 410) [a score indicating a degree of abnormality calculated by the first model]. The data classification manager may also determine a cohesion and a separation [a score indicating a degree of abnormality calculated by the first model] for the respective set of filtered outliers for each of the classification models in the ensemble (step 412). For example, in one embodiment, determining the cohesion and the separation for the respective set of filtered outliers for each of the classification models in the ensemble may include determining a unified measure [a score indicating a degree of abnormality calculated by the first model] of the cohesion and the separation for the respective set of filtered outliers for each of the classification models in the ensemble, wherein the unified measure of the cohesion and the separation is a value in a range from -1 to 1… In another example embodiment, detecting the novel class in response to the threshold number of the plurality of classification models in the ensemble determining that the novel class is detected may include detecting the novel class in response to determining, by each of a threshold number of the plurality of classification models in the ensemble, that a threshold amount of the respective set of filtered outliers has a positive unified measure of the cohesion and the separation [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value]…)
Regarding claims 7 and 8, the limitations are similar to claim 1 limitations and thus rejected under the same rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8 are rejected under 35 U.S.C. 103 as being unpatentable over Hearty et al. (US 20200280578, hereinafter ‘Hearty’) in view of Suzani et al. (US 20220188694, hereinafter ‘Su’)
Regarding independent claim 1, Hearty teaches a determination device comprising: processing circuitry configured to: (in [0010] In addition, it should be understood that embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored on non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and/or application specific integrated circuits (“ASICs”)…)
determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data; (As depicted in Fig. 12; And in [0091] FIG. 12 is a flowchart that illustrates an OAO feature drift hardening process 1200, according to embodiments described herein. FIG. 12 is described with respect to the fraud prevention server 1135 of FIG. 11… [0094] The alerting component 1210 may be a software service that distributes alerts related to feature drift and model evaluation performance to relevant groups including individuals that maintain the models and users of the models. The OAO drift weighting component 1212 may be a software module that enables the setting of manually or automatically derived drift weights [and information regarding a second model generated based on the over-detection data], such as event recency weights and the application of said weights as a feature within models. The OAO model set 1214 may be a collection of subcomponents including: 1) longer-term models 1218, intended to learn longer-term trends from mostly time-series features or alternatively to model drift components using a subset of information such as drift weighting and recency features, and 2) shorter-term models 1220 [alternatively the plurality of models including information regarding a second model generated based on the over-detection data], intended to learn short-term state from mostly stationary features without the impact of long-term drift. [0095] The OAO model retraining component 1216 receives drift monitoring results from the OAO drift monitoring component 1208 [information regarding over-detection data over-detected by an abnormality detection system that uses the first model] and model monitoring results from the OAO model evaluation and monitoring component 1226 [determine necessity of relearning of a first model based on at least one of information regarding the first model generated by learning initial learning data known to be normal, information regarding over-detection data over-detected by an abnormality detection system that uses the first model]. The OAO model retraining component 1216 [determine necessity of relearning of a first model] outputs one or more retrained models to the OAO model set 1214 in response to determining that retraining is necessary based on at least one of the drift monitoring results or the model monitoring results.)
and notify of a result of a determination. (in [0094] The alerting component 1210 may be a software service that distributes alerts [notify of a result of a determination] related to feature drift and model evaluation performance to relevant groups including individuals that maintain the models and users of the models…; And in [0097] The OAO model evaluation and monitoring component 1226 may be a software component designed to assess trained and live model performance through statistical evaluation, compare the results of evaluation to defined performance requirements, and trigger an alert [notify of a result of a determination] when the comparison determines that the performance deviates from requirement thresholds. Lastly, the OAO model output visualization component 1228 may be designed to support analysis and model diagnostic activity by individuals that maintain the models or users of the models.)
One of ordinary skill in the art would understand that concept drift as an abnormality detection mechanism as noted above.
Additionally, Su expressly teaches concept drift as an abnormality detection mechanism, in [0017] The present invention relates to model decay of an anomaly detector due to concept drift. Herein are machine learning techniques for dynamically self-tuning an anomaly score threshold. Herein is a dynamic thresholding approach to preserve an anomaly detector's ability to accurately classify data items even when concept drift causes a distribution of data in a production environment to diverge from training data. A goal is to make the anomaly detector able to adapt to changes in value range and distribution trends of input data. Herein is anomaly score normalization and dynamic thresholding that are proven to be effective in detecting and automatically handling concept drifts in data of various applications [information regarding over-detection data over-detected by an abnormality detection system that uses the first model] such as structured log data of cloud servers. [0019] Unlike other techniques, dynamic thresholding entails a probabilistic approach that normalizes anomaly scores and rescales them to a range [0,1], where 0 indicates a certainly normal item and 1 indicates a certain anomaly. Then, a dynamic threshold based on statistical measures of the normalized anomaly scores is applied. As concept drift happens over time […information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data], the system automatically raises the threshold based on distribution of anomaly scores […information regarding over-detection data over-detected by an abnormality detection system that uses the first model, and information regarding a second model generated based on the over-detection data]. This lowers a false positive rate and makes the system more resilient.
Su and Hearty are analogous art because both involve developing information retrieval and data processing techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for retrieving information for anomaly detection of log data based on dynamic thresholding and drift monitoring for detecting suspicious anomalous activities, as disclosed by Su with the method of developing information retrieval and processing techniques to monitor abnormality in fraud detection as disclosed by Hearty.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Su and Hearty as noted above. Doing so allows a system to continue detecting only the most suspicious anomalous activities; that helps keep cloud server machines safe from malfunctions and internal and external attacks, (Su, 0024).
Regarding claim 2, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value. (in [0093] The feature calculator 1206 may be a software module that is used to generate “features” (data input variables) for modeling. The OAO drift monitoring component 1208 may be a software module that measures the degree of feature drift [a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data] across individual features and combinations of 2 . . . n features (where n is the total number of features in the set, for example, some or all of the behavioral features described above). These combinations may be problem specific or automatically defined (e.g., every combination of features may be assessed). The OAO drift monitoring component 1208 may also compare drift against a defined threshold and trigger an alert and retraining activity [wherein the processing circuitry is further configured to determine that the relearning of the first model…] when the comparison determines that the drift exceeds a certain threshold [wherein the processing circuitry is further configured to determine that the relearning of the first model is necessary when a ratio of the number of pieces of the over- detection data to the number of pieces of the initial learning data exceeds a predetermined value].)
Regarding claim 3, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary in a case where a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value. (in [0105] The fraud prevention server 1135 executes the OAO model retraining component 1216 to consume information generated by the OAO drift and model monitoring components 1208 and 1226, and identify when a retraining of the model is necessary. This decision is based on statistical analysis [determine that the relearning of the first model is necessary in a case where a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard…] of the output of the OAO model evaluation and monitoring component 1226. Model retraining [determine that the relearning of the first model is necessary…] may be initiated when either model score distribution (e.g. score central tendency, proportion of traffic identified as high risk) begins to deviate beyond accepted levels [… a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value], when the fraud prevention server 1135 executes the OAO drift monitoring component 1208 and identifies a consistently higher degree of drift, or when a defined period of time has passed (e.g., one week, one month, three months, or some other temporal period) [… a ratio of the number of types of over- detection data when the over-detection data is classified into a plurality of types based on a predetermined standard, to the number of types of initial learning data when the initial learning data is classified based on the standard exceeds a predetermined value]. In addition, model retraining may also be manually initiated.)
Regarding claim 4, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a loss function of the second model exceeds a predetermined value. ([0093] The feature calculator 1206 may be a software module that is used to generate “features” (data input variables) for modeling. The OAO drift monitoring component 1208 may be a software module that measures the degree of feature drift [determine that the relearning of the first model is necessary when a loss function…] across individual features and combinations of 2 . . . n features (where n is the total number of features in the set, for example, some or all of the behavioral features described above). These combinations may be problem specific or automatically defined (e.g., every combination of features may be assessed). The OAO drift monitoring component 1208 may also compare drift against a defined threshold and trigger an alert and retraining activity when the comparison determines that the drift exceeds a certain threshold [determine that the relearning of the first model is necessary when a loss function of the second model exceeds a predetermined value].)
Regarding claim 5, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value. (in [0105] The fraud prevention server 1135 executes the OAO model retraining component 1216 to consume information generated by the OAO drift and model monitoring components 1208 and 1226, and identify when a retraining of the model is necessary. This decision is based on statistical analysis [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value.] of the output of the OAO model evaluation and monitoring component 1226. Model retraining [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system…] may be initiated when either model score distribution (e.g. score central tendency, proportion of traffic identified as high risk) begins to deviate beyond accepted levels [determine that the relearning of the first model is necessary when a ratio of data in which an abnormality is not detected by the abnormality detection system using the second model among detection target data exceeds a predetermined value], when the fraud prevention server 1135 executes the OAO drift monitoring component 1208 and identifies a consistently higher degree of drift, or when a defined period of time has passed (e.g., one week, one month, three months, or some other temporal period). In addition, model retraining may also be manually initiated.)
Regarding claim 6, the rejection of claim 1 is incorporated and Hearty further teaches the determination device according to claim 1, wherein the processing circuitry is further configured to (in [0029] And in [0031])
determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value. (in [0096] The OAO model selector 1222 may be a mathematical function designed to select which OAO models to execute against a sample based on the observed drift of the sample [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value]. The score resolution component 1224 may be a software component designed to combine the scores from various OAO models into a single result. The combination of the scores from various OAO models into the single result may be achieved with a problem-specific regression function, or using ensemble resolution techniques such as stacking or bucketing. [0097] The OAO model evaluation and monitoring component 1226 [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value] may be a software component designed to assess trained and live model performance through statistical evaluation, compare the results of evaluation to defined performance requirements, and trigger an alert when the comparison determines that the performance deviates from requirement thresholds [a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value]… [0105] The fraud prevention server 1135 executes the OAO model retraining component 1216 to consume information generated by the OAO drift and model monitoring components 1208 and 1226 [determine that the relearning of the first model is necessary when a score indicating a degree of abnormality calculated by the first model exceeds a predetermined value], and identify when a retraining of the model is necessary. This decision is based on statistical analysis of the output of the OAO model evaluation and monitoring component 1226. Model retraining may be initiated when either model score distribution (e.g. score central tendency, proportion of traffic identified as high risk) begins to deviate beyond accepted levels, when the fraud prevention server 1135 executes the OAO drift monitoring component 1208 and identifies a consistently higher degree of drift, or when a defined period of time has passed (e.g., one week, one month, three months, or some other temporal period)…)
Regarding claims 7 and 8, the limitations are similar to claim 1 limitations and thus rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Phadke et al. (US 20190081969): teaches anomaly detection is an important aspect in automated data analysis. Anomalies in general may be understood as instances of data that lie outside of a normal or expected range or threshold. A deviation of data from its normal or expected values can be indicative of an underlying problem in a service provider system. Statistical anomaly detection techniques are known, robust computational approaches that aim to statistically model a physical phenomenon and capture when data representing the physical phenomenon statistically deviates from the normal. A statistical detection system typically generates an alert that indicates an anomaly, i.e., a deviation in a data value that may be a precursor to a larger underlying issue in the service provider system.
Poli et al. (US 20220027750): teaches techniques for identifying real-world behavior. A machine learning model can be trained to learn patterns of real-world behavior and may use the learned pattern to perform predictions (e.g., risk predictions, etc.). For example, an online service provider that receives transaction requests (e.g., login requests, content access requests, payment requests, purchase requests, etc.) may train one or more machine learning models to recognize behavior patterns of legitimate transaction attempts and behavior patterns of fraudulent transaction attempts.
Wang et al. (US 20080071721): teaches
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWATOSIN ALABI whose telephone number is (571)272-0516. The examiner can normally be reached Monday-Friday, 8:00am-5:00pm EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWATOSIN ALABI/Primary Examiner, Art Unit 2129