DETAILED ACTION
This correspondence is responsive to the Application filed on October 2, 2023. Claims 1-17, 19, and 36-37 are pending in the case, with claims 1, 19 and 36-37 in independent form. Claims 18, 20-35, and 38 are cancelled.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Summary of Detailed Action
I. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
II. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
III. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
IV. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
V. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
VI. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
VII. Claim 13 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
Claim 36 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter and does not fall within at least one of the four categories of patent eligible subject matter.
Claims 1-3, 7-9, 11, 14, 17, 19 and 37 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim(s) 1, 19, and 36-37 are rejected under 35 U.S.C. 103 as being unpatentable over Elprin et al. (Pub. No. US 2021/0133632 A1, filed November 3, 2020) hereinafter Elprin, in view of Maughan et al. (Pub. No. US 2017/0330109 A1, published November 16, 2017) hereinafter Maughan.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Elprin in view of Maughan, and further in view of Patton et al.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Elprin in view of Maughan, and further in view of Nasr-Azadani et al. (Pub. No. US 2021/0224696 A1, filed January 20, 2021) hereinafter Nasr-Azadani.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
I. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 19, 36 and 37 include the limitations for monitoring “an input data stream” of the ML system, wherein the monitoring includes: a first monitoring for detecting data drift based on a distribution change of” the input data” and for determining a type and a range of data drift. It is not clear if “the input data” is the same as the “input data stream” or if “the input data” is not the same as the “input data stream.” It is further unclear if the input data is only part of the input data stream or if the input data is all of the input data stream. It is yet further unclear if the input data stream is streaming input data or if the input data stream is any data being read into the ML system. Applicant may cancel claims 1, 19, 36 and 37 or amend claims 1, 19, 36 and 37 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2-17 depend from claim 1 and are rejected for the same reasons discussed above with respect to claim 1.
II. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 19, 36 and 37 include the limitations that inconsistently recite the terms “performance metrics requirements” and “performance metrics requirements obtained.” It is not clear if “the performance metrics requirements” are the same as “the performance metrics requirements obtained” or if “the performance metrics requirements” are different than “the performance metrics requirements obtained.” Applicant may cancel claims 1, 19, 36 and 37 or amend claims 1, 19, 36 and 37 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2-17 depend from claim 1 and are rejected for the same reasons discussed above with respect to claim 1.
III. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 19, 36 and 37 include the limitations for obtaining performance metrics requirements used “for data drift handling” and applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model “to handle the data drift.” It is not clear what data drift handling or to handle the data drift means. For example, is handling data drift mean that the model processes data drift? Or does handling the data drift mean that the model processes the data drift at a required performance metric? Or does handling the data drift mean something else entirely? Applicant may cancel claims 1, 19, 36 and 37 or amend claims 1, 19, 36 and 37 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2-17 depend from claim 1 and are rejected for the same reasons discussed above with respect to claim 1.
IV. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 19, 36 and 37 include the limitations for “if the first monitoring and the second monitoring both detect data drift, selecting from a data repository storing a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift; testing the selected data drift adaptor to determine if the performance metrics requirements are met; and if the performance metrics requirements are met, applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift.” If the first and second monitoring do not both detect data drift then there is no selected data drift adapter to test and it is unclear what the selected adapter is or how it is tested when both first and second monitoring do not detect data drift. It is further unclear to test the selected data drift adapter and apply the selected data drift adapter when the performance metrics are met if the first and second monitoring do not both detect drift. Applicant may cancel claims 1, 19, 36 and 37 or amend claims 1, 19, 36 and 37 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2-17 depend from claim 1 and are rejected for the same reasons discussed above with respect to claim 1.
V. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 19, 36 and 37 include the limitations for “selecting from … a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift; testing the selected data drift adaptor to determine if the performance metrics requirements are met.” It is not clear what a data drift adapter is or is not. For example, is a data drift adapter a processor or system or software or any process or training data that is used and applied for adapting a model to a data drift? Or is a data drift adapter a parameter or hyperparameter or weight or bias or coefficient used to tune, fine-tune, optimize and adapt the model to a data drift? Or is a data drift adapter something else entirely? Thus, it is not clear what the data drift adapter is or is not and the boundaries of the claim limitation are indefinite. It is further unclear how to select a data drift adapter based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift because it is not clear what the data drift adapter is or is not, much less how one is selected based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift. It is yet still further unclear how to test the selected data adapter because it is not clear what the selected adapter is or is not, much less how to test the selected data drift adapter to determine if the performance metrics requirements are met. Applicant may cancel claims 1, 19, 36 and 37 or amend claims 1, 19, 36 and 37 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2-17 depend from claim 1 and are rejected for the same reasons discussed above with respect to claim 1.
VI. Claims 1-17, 19 and 36-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 19, 36 and 37 include the limitations for “testing the selected data drift adaptor to determine if the performance metrics requirements are met; and if the performance metrics requirements are met, applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift.” It is not clear what how to test the selected data drift adapter and determine if the performance metrics are met and only apply the selected data drift adapter if the performance metrics are met because testing the data drifter adapter implies that the data drift adapter is somehow applied to the first trained ML model in order to test the data drift adapter and determine if the performance metrics are met or not met. Thus, it not clear how the selected data drift adapter is ether only applied to the trained ML model after it has already been determined that the performance metrics are met. Or is the selected data drift adapter reapplied after determining that the performance metrics are met? Or, is the selected data drift adapter not applied as all to the first trained ML model and only applied to a different model or a copy or a copy of the first trained ML model? Thus, the limitation is not clear and claims 1, 19, 36 and 37 are indefinite. Applicant may cancel claims 1, 19, 36 and 37 or amend claims 1, 19, 36 and 37 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2-17 depend from claim 1 and are rejected for the same reasons discussed above with respect to claim 1.
VII. Claim 13 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 13 depends from claim 10 and recites a policy function that includes a theta variable
PNG
media_image1.png
28
28
media_image1.png
Greyscale
that is not specified as to what the variable represents. For example, does the variable represent an angle or a some other parameter or something else entirely? Applicant may cancel claim 13 or amend claim 13 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 36 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because independent claim 36 recites “A network node configured for automated handling of data drift in a network using a machine learning (ML) system according to claim 19.” The network node configured for automated handling of data drift in a network using a machine learning (ML) system can reasonably be interpreted as software. Therefore, the network node is software and does not fall within at least one of the four categories of a process, machine, manufacture, or composition of matter.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-3, 7-9, 11, 14, 17, 19 and 37 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) subject matter at a general, high-level of a method for handling of data drift, the method comprising: monitoring an input data stream of the ML system, wherein the monitoring includes: a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift; and a second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model; if the first monitoring and the second monitoring both detect data drift, selecting from a data repository storing a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift; testing the selected data drift adaptor to determine if the performance metrics requirements are met; and if the performance metrics requirements are met, handle the data drift, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the recitations of a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift; and a second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model; testing the selected data drift adaptor to determine if the performance metrics requirements are met; are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I). This judicial exception is not integrated into a practical application and the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claims 1-17, 19 and 37 recite one of the four statutory categories of patent able subject matter and belong to the statutory class(es) of a process (method claims 1-17), a machine (system/apparatus claims 19), and an article of manufacture (non-transitory computer readable media claims 37).
The Examiner notes that if claim 36 is amended only to recite one of the four categories of a process, machine, manufacture, or composition of matter, then claim 36 would be rejected as being directed to an abstract idea without significantly more, similar to the rejections of independent claims 1, 19, and 36.
Claim 1 recites a method, thus a process and one of the four statutory categories of patentable subject matter. However, claim 1 further recites for handling of data drift, the method comprising: monitoring an input data stream of the ML system, wherein the monitoring includes: a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift; and a second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model; if the first monitoring and the second monitoring both detect data drift, selecting from a data repository storing a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift; testing the selected data drift adaptor to determine if the performance metrics requirements are met; and if the performance metrics requirements are met, handle the data drift, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the recitations of a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift; and a second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model; testing the selected data drift adaptor to determine if the performance metrics requirements are met; are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
computer-implemented (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
automated (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
in a machine learning (ML) system including a plurality of trained ML models (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
obtaining performance metrics requirements used for data drift handling (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31.).
applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and transmitting data over a network is well-understood, routine and conventional (MPEP 2106.05(d), and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 2, dependent on claim 1, recites additional abstract ideas for wherein the testing includes: determining an amount of the input data to collect; using the collected input data as input and determining whether the performance metrics requirements are met based on output values, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the recitation of using the collected input data as input and determining whether the performance metrics requirements are met based on output values are also mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
collecting the determined amount of input data (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31.).
to a second trained ML model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
applying the selected data drift adaptor to the second trained ML model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
from the second trained ML model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 3, dependent on claim 1, recites only additional abstract ideas for wherein the performance metrics requirements include one or more of: a maximum time used for drift adaptation, a maximum amount of resources allocated for drift adaptation, a maximum amount of dataset size used for drift adaptation, and an ML model target accuracy tolerance range, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, these recitations of wherein the performance metrics requirements include one or more of: a maximum time used for drift adaptation, a maximum amount of resources allocated for drift adaptation, a maximum amount of dataset size used for drift adaptation, and an ML model target accuracy tolerance range are also mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I)).
Claim 7, dependent on claim 1, recites additional abstract ideas for wherein the second monitoring for detecting data drift based on a drop in accuracy of the first trained ML model includes using one or more of: Cumulative Sum (CUSUM) method, which are mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
Adaptive Window (ADWIN) method, Early Drift Detection Method (EDDM), and Fast Hoeffding Drift Detection Method (FHDDM) (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 8, dependent on claim 1, recites additional abstract ideas for wherein selecting from the data repository one of the data drift adaptors based on the performance metrics requirements obtained, the type of the first trained ML model, and the determined type and range of data drift, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the recitation of using the collected input data as input and determining whether the performance metrics requirements are met based on output values are also mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
using reinforcement learning (RL) (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 9, dependent on claim 8, recites additional abstract ideas for and the selecting includes: defining a first state and a second state for target accuracy, wherein, in the first state, the target accuracy is within a predetermined target accuracy tolerance range and, in the second state, the target accuracy is not within the predetermined target accuracy tolerance range; defining actions of selecting each one of the data drift adaptors from the plurality of data drift adaptors in the data repository; defining a reward function based on the target accuracy, time consumption, and resource consumption of the selected one of the data drift adaptors; and defining a policy function based on a probability distribution of taking each action, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the recitations of defining a reward function based on the target accuracy, time consumption, and resource consumption of the selected one of the data drift adaptors; and defining a policy function based on a probability distribution of taking each action, are also mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the RL used includes policy-based RL (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 11, dependent on claim 8, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the RL used includes Q-learning (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 14, dependent on claim 2, recites additional abstract ideas for wherein determining an amount of the input data to collect and the determining includes: initializing a data size range; initializing states, each state corresponding to one candidate data size of a plurality of candidate data sizes within the data size range; initializing a first action of increasing the data size and a second action of decreasing the data size; and defining a reward function based on a gained accuracy using collected data and target accuracy, which are mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
includes using reinforcement learning (RL) (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 17, dependent on claim 14, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein the RL includes Q-learning (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 19 recites a system, thus a machine and one of the four statutory categories of patentable subject matter. However, claim 19 further recites for handling of data drift monitor an input data stream of the ML system, wherein the monitoring includes: a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift; and a second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model; if the first monitoring and the second monitoring both detect data drift, select from a data repository storing a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift; test the selected data drift adaptor to determine if the performance metrics requirements are met; and if the performance metrics requirements are met, handle the data drift, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the recitations of a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift; and a second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model; test the selected data drift adaptor to determine if the performance metrics requirements are met; are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
A machine learning system comprising: processing circuitry; and a memory containing instructions executable by the processing circuitry for (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
automated (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
obtain performance metrics requirements used for data drift handling (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31.).
apply the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and transmitting data over a network is well-understood, routine and conventional (MPEP 2106.05(d), and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 37 recites a product comprising a non-transitory computer readable medium that parallels the method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 37. Accordingly, claim 37 is rejected based on substantially the same rationale as set forth above with respect to claim 1.
More specifically, claim 37 recites a product comprising a non-transitory computer readable medium, thus an article of manufacture and one of the four statutory categories of patentable subject matter. However, claim 37 further recites to perform the method of claim 1, which as discussed above is directed to mental processes and mathematical concepts and does not include any additional elements which integrate the abstract idea into a practical application. Please see above with respect to the rejection of claim 1 as directed to mental processes and mathematical concepts and does not include any additional elements which integrate the abstract idea into a practical application. More specifically regarding the additional element of A computer program product comprising a non-transitory computer readable medium storing a computer program comprising instructions which, when executed by processing circuitry, causes the processing circuitry to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and transmitting data over a network is well-understood, routine and conventional (MPEP 2106.05(d), and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 19, and 36-37 are rejected under 35 U.S.C. 103 as being unpatentable over Elprin et al. (Pub. No. US 2021/0133632 A1, filed November 3, 2020) hereinafter Elprin, in view of Maughan et al. (Pub. No. US 2017/0330109 A1, published November 16, 2017) hereinafter Maughan.
Regarding claim 1, Elprin teaches:
A computer-implemented method for automated handling of data drift in a machine learning (ML) system including a plurality of trained ML models, the method comprising (i.e., In an aspect, a system for monitoring machine learning models is provided (A computer-implemented method for automated handling of data drift in a machine learning (ML) system). Elprin, Figs 4-8, para 10, 8, 37. Systems and methods of the present disclosure may be capable of detecting data drift and/or performance drift (e.g., changes in data and model performance), performing integrity checks, notifying users/applications of a change or detected events, and/or automatically retraining models (trained models) and re-scoring models. In some cases, the provided systems and methods may also be capable of monitoring traffic and a portfolio of models (including a plurality of trained ML models,) over time. In some cases, the provided systems and methods may monitor data drift or performance of a model in different phases (e.g., development, deployment, prediction, validation, etc) or perform data integrity checks for models that have been deployed in a development, test, or production environment. Elprin, Figs 4-8, para 8, 10, 37):
obtaining performance metrics requirements used for data drift handling (i.e., In some embodiments, the drift of the plurality of machine learning models comprises a drift in a dataset or a drift in model performance. In some embodiments, the first GUI is configured to receive a user input indicative of test method for an input feature, a threshold for triggering a detection event, or a combination of multiple metrics for multiple input features for a machine learning model selected from the plurality of machine learning models (obtaining performance metrics requirements (performance metrics, thresholds, rules para 12 requirements) used for data drift handling). In some cases, the first GUI is configured to further display the threshold over a diagram of the drift. Elprin, Figs 4-7, para 11, 12, 13-16, 34, 48. 69.);
monitoring an input data stream of the ML system, wherein the monitoring includes (i.e., Systems and methods provided herein can efficiently monitor models at scale. In particular, systems and methods provide a visual tool for monitoring models with improved accuracy. Various aspects of models are monitored by the provided model monitor systems and methods. Systems and methods of the present disclosure may be capable of detecting data drift and/or performance drift (e.g., changes in data and model performance) (monitoring an input data stream of the ML system, wherein the monitoring includes), performing integrity checks, notifying users/applications of a change or detected events, and/or automatically retraining models and re-scoring models. In some cases, the provided systems and methods may also be capable of monitoring traffic and a portfolio of models over time. In some cases, the provided systems and methods may monitor data drift or performance of a model in different phases (e.g., development, deployment, prediction, validation, etc) or perform data integrity checks for models that have been deployed in a development, test, or production environment. Elprin, Figs 4-8, para 8, 46-48, 73):
a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift (i.e., In some cases, systems and methods herein may permit users to set up rules for detecting the model drift. For example, one or more rules may be defined for detecting (input) data drift such as by checking descriptive statistics, data types (e.g., strings, integers, etc.), data ranges, and data sparsity/missing data, and then comparing them to the original training data. Drastic changes in input data distributions may indicate serious model degradation (a first monitoring for detecting data drift based on a distribution change of the input data and for determining a type and a range of data drift). The rules may also include metrics defined to track the difference between data used to train the model versus data that are being presented to the model to score. If the difference crosses a threshold (determining if amount of difference crosses a threshold range of data drift) or is drifting significantly (determining a significant type of data drift), that is a strong indicator of model drift and degradation. The provided systems and methods may allow users to customize/add any rules (e.g., drift distribution analysis) that are suitable for monitoring the input data presented to a model in order to see if they have changed. Elprin, Figs 4-8, para 48, 49, 46, 85, 87); and
a second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model (i.e., Systems and methods herein may detect and track data drift in the model's input features and output predictions. Users may be permitted to select among a variety of statistical checks to suit different monitoring needs. For example, users may supply proprietary ground truth data for a model, and systems herein may ingest it to calculate and track the model's prediction quality using one or more standard measures such as accuracy (a second monitoring for detecting data drift based on a drop in accuracy (detecting data drift based on measuring accuracy difference, drop) of a first trained ML model), precision, and the like. The system may automatically check if the characteristics of the predictions versus the target data that were used to train the model are significantly different. The system may also track the difference between data that were used to train the models verses the data that are being presented to the models to score. Systems herein may permit users to set up scheduled checks, use APIs to ingest data, and configure alert notification recipients such users are capable of monitoring models at a large scale in a standardized manner with improved efficiency. Elprin, Figs 4-8, para 47, 46, 73-74, 80. The model monitor system may be configured to perform data integrity checks and detect data drift and accuracy degradation (a second monitoring for detecting data drift based on a drop in accuracy (accuracy degradation, drop) of a first trained ML model). Elprin, Figs 4-8, para 74, 73, 80, 46-47.);
if the first monitoring and the second monitoring both detect data drift (i.e., Multiple aspects of models may be monitored along with time. In some cases, performance of a model, accuracy of a model, data used in different phases (e.g., development, deployment, prediction, validation, etc) may be monitored along with time. The model monitor system may be capable of monitoring model drift. A model drift may include model performance drift and/or data drift (if the first monitoring and the second monitoring both detect data drift (monitoring, detecting both model performance drift and data drift)). A model drift may be defined as a drift in model performance, data, or others with respect to a reference time (e.g., initial time), a normal condition/value, an initial model or initial dataset, or other user defined references. A model drift may refer to change in data or performance of a model over time. Elprin, Figs 4-8, para 46, 47, 74, 73. 80, 8), selecting from a data repository storing a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift (i.e., In some cases, upon receiving a notification, the model monitor system may take one or more actions. The one or more actions may include, for example, unpublish model/fallback to last known working version of the model, return error/fallback response values for models that are problematic, or retrain and re-deploy models for consumption by different applications (selecting from a data repository storing a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift (upon notification model monitoring system automatically takes action and selects from a data repository storing a plurality of model versions, model retrain data drift adapters one of the model versions, retrain data drift adapters based on the notification rules performance metrics thresholds requirements obtained, a notification model version type of the first trained ML model and the notification determined type and range of data drift)). Elprin, Figs 1-15, para 82, 87, 8, 52, 82);
The Examiner notes that the claim limitation of “if the first monitoring and the second monitoring both detect data drift” is a condition precedent and the limitation of “selecting from a data repository storing a plurality of data drift adaptors one of the data drift adaptors based on the performance metrics requirements obtained, a type of the first trained ML model, and the determined type and range of data drift” would be a contingent limitation if the claimed invention does not require the condition to occur. However, the claimed invention limitation for testing the selected data drift adapter to determine if the performance metrics are met appears to require that the data drift adapter was selected, which in turn appears to require that the condition of the first monitoring and second monitoring both detect drift appears to be required to occur.
See also MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim.
However, for the system claim the claimed structure must be present in the system regardless of whether the condition is met and the function is actually preformed, which is the case for system claim 19.
The examiner further notes that the claim limitations at issue are unclear and are rejected as being indefinite as discussed above with respect to the rejection claims 1-17, 19 and 36-37 under 35 U.S.C. 112(b).
testing the selected data drift adaptor to determine if the performance metrics requirements are met; and
if the performance metrics requirements are met, applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift.
Elprin teaches the selected drift adapter, performance metrics requirements, first trained ML model to handle the data drift. Elprin does not specifically disclose testing the selected data drift adaptor to determine if the performance metrics requirements are met; and if the performance metrics requirements are met, applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift.
However, Maughan teaches in the field related to predictive analytics and more particularly relates to drift detection and correction for predictive analytics. Maughan, para 2. Maughan, which is analogous to the claimed invention because Maughan is directed to drift correction, teaches that, Methods are presented for drift detection and correction for predictive analytics. In one embodiment, a method includes generating one or more predictive results by applying a model to workload data. In certain embodiments, workload data may include one or more records. In further embodiments, a model may include one or more learned functions based on training data. In one embodiment, a method includes detecting a drift phenomenon relating to one or more predictive results. In a certain embodiment, a method includes retraining a model based on updated training data, in response to detecting a drift phenomenon. Maughan, para 7, 84. [0084] The retrain module 302, in one embodiment, performs one or more tests on retrained machine learning (testing the selected data drift adaptor to determine if the performance metrics requirements are met), to determine whether predictions from the retrained machine learning are more accurate than from the original machine learning. For example, the retrain module 302 may perform A/B testing, using both the original machine learning and the retrained machine learning for a predefined period after retraining the machine learning, alternating between the two, randomly selecting one or the other, and/or providing predictions or other results from both the original model and the retrained model to a user or other client 104. The retrain module 302 may perform the testing for a predefined trial period, then may select the more accurate machine learning (if the performance metrics (more accurate) requirements are met, applying the selected data drift adaptor (applying retraining drift adapter) to the first trained ML model to adapt the first trained ML model to handle the data drift (to adapt, retrain, and use the trained ML model to accurately handle drift, see also para 7)), may allow a user or other client 104 to select one of the original model and the retrained model, or the like for continued use. Maughan, Figs 3, 8-9, para 84, 7, 49-50, 68, 72, 74-79, 83, 94, 98, 162,-163.
The Examiner notes that the claim limitation of “if the performance metrics requirements are met” is a condition precedent and the limitation of “applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift” would be a contingent limitation if the claimed invention does not require the condition to occur. However, the claimed invention limitation for applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift appears to be required to occur. The examiner further notes that the claim limitations at issue are unclear and are rejected as being indefinite as discussed above with respect to the rejection claims 1-17, 19 and 36-37 under 35 U.S.C. 112(b).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the model monitoring for handling data drift, selected drift adapter, performance metrics requirements, first trained ML model to handle the data drift of Elprin using the feature for testing the selected data drift adaptor to determine if the performance metrics requirements are met, and if the performance metrics requirements are met, applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift of Maughan, with a reasonable expectation of success, in order to provide for testing and selecting the most accurate model for continued use, because changes in data may cause a predictive model to become less accurate over time, even if the predictive model was initially accurate. Maughan, para 5, 84.
This would have provided the advantages of using an improved model with more accurate performance.
Claim 19 recites a system that parallels the method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 19. Accordingly, claim 19 is rejected based on substantially the same rationale as set forth above with respect to claim 1. More specifically regarding A machine learning system comprising: processing circuitry; and a memory containing instructions executable by the processing circuitry for … the machine learning system operative to (Elprin, Fig 15, para 98-111).
Claim 36 recites a network node that parallels the method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 36. Accordingly, claim 36 is rejected based on substantially the same rationale as set forth above with respect to claim 1. More specifically regarding A network node, in a network (Elprin, Fig 15, para 98-111).
Claim 37 recites a computer program product that parallels the method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 37. Accordingly, claim 37 is rejected based on substantially the same rationale as set forth above with respect to claim 1. More specifically regarding A computer program product comprising a non-transitory computer readable medium storing a computer program comprising instructions which, when executed by processing circuitry, causes the processing circuitry to perform (Elprin, Fig 15, para 98-111)
Claim(s) 2 is rejected under 35 U.S.C. 103 as being unpatentable over Elprin in view of Maughan as applied to claim 1 above, and further in view of Patton et al. (Pub. No. US 2019/0171428 A1, published June 6, 2019) hereinafter Patton.
Regarding claim 2, which depends from claim 1 and recites:
wherein the testing includes: determining an amount of the input data to collect; collecting the determined amount of input data; using the collected input data as input to a second trained ML model; applying the selected data drift adaptor to the second trained ML model; and determining whether the performance metrics requirements are met based on output values from the second trained ML model.
Elprin in view of Maughan teaches the method of claim 1 from which claim 2 depends, including testing, applying the selected data drift adaptor, a trained ML model, determining whether the performance metrics requirements are met. Elprin in view of Maughan does not explicitly disclose determining an amount of the input data to collect; collecting the determined amount of input data; using the collected input data as input to a second trained ML model; applying the selected adaptor to the second trained ML model; and determining whether the performance metrics requirements are met based on output values from the second trained ML model.
However, Patton teaches in the field related to the computing field, and more specifically to a new and useful method for model management in the computing field. Patton, para 3. Patton, which is analogous to the claimed invention because Patton is directed to model management and concept drift detection, teaches that, [0126] However, the training data set can be otherwise generated. [0127] Generating data sets can optionally include generating testing data sets, which can be used to test the trained model (examples shown in FIG. 3 and FIG. 6). The testing data set can be generated concurrently or asynchronously from training data set generation, and can use the same or different source pool. The testing data set can have the same or different parameters (e.g., size, distribution, etc.) as the training data set (determining an amount (determining a same or different size) of the input data (source input data) to collect (use, collect); collecting the determined amount (collecting using the same of different determined size) of input data (source input data) using the collected input data as input to a second trained ML model; applying the selected data drift adaptor to the second trained ML model; and determining whether the performance metrics requirements are met based on output values from the second trained ML model);. In a first variation, the testing data set is generated through the third variant for training data set generation. In a second variation, the testing data set is generated using one of the methods disclosed above for training data set generation. However, the testing data set can be otherwise generated. [0128] Validating the candidate model S300 functions to measure the performance of the candidate models (using the collected input data as input to a second trained ML model (using the collected input source data as input to a second candidate trained ML model); applying the selected data drift adaptor to the second trained ML model (applying the selected validation adapter to the second candidate trained model); and determining whether the performance metrics requirements are met based on output values from the second trained ML model (measure the performance to determine if performance metrics meet validation performance requirements based on output values from the second candidate trained ML model)). S300 is preferably performed by the validation system, but can be otherwise performed. S300 is preferably performed after S200, but can alternatively be performed after a threshold number of challengers for the deployed model have been generated, or be performed at any suitable time. S300 is preferably automatically performed, but can alternatively be manually performed. S300 is preferably performed using the trained candidate model (output by S200) and a training data set, but can alternatively be performed using a real data set (e.g., representative of data fed to the corresponding deployed model), using a second model (e.g., a higher-accuracy model run on the same test data), or performed using any other suitable data. The generated metrics are preferably one or more of the evaluation metrics, but can alternatively be other metrics. Patton, Figs 3, 6, para 126-128, 129-130. Deploying the candidate model into a production environment S400 functions to run the candidate model on real world data, and can optionally replace the prior model. In variants, this can function to deploy a new model that is better suited to detecting events, despite the concept drift. The candidate model is preferably deployed when the deployment conditions (criteria) are met, but can be deployed at any suitable time. Patton, Figs 3, 6, para 130, 129, 126-128.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the model monitoring for handling data drift, selected drift adapter, performance metrics requirements, first trained ML model to handle the data drift of Elprin using the feature for testing the selected data drift adaptor to determine if the performance metrics requirements are met, and if the performance metrics requirements are met, applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift of Maughan and the determining an amount of the input data to collect; collecting the determined amount of input data; using the collected input data as input to a second trained ML model; applying the selected adaptor to the second trained ML model; and determining whether the performance metrics requirements are met based on output values from the second trained ML model of Patton, with a reasonable expectation of success, in order to provide for testing and selecting the most accurate model for continued use, because changes in data may cause a predictive model to become less accurate over time, even if the predictive model was initially accurate and in order to provide a system for model management that includes generating models for subsequent data analyses, such as detections, predictions, and decision making, and gathering training data, building new models, evaluating the new models. Maughan, para 5, 84. Patton, para 4, 5, 126-130.
This would have provided the advantages of using an improved model with more accurate performance and the advantages of automating model management, generation and evaluation.
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Elprin in view of Maughan as applied to claim 1 above, and further in view of Nasr-Azadani et al. (Pub. No. US 2021/0224696 A1, filed January 20, 2021) hereinafter Nasr-Azadani.
Regarding claim 7, which depends from claim 1 and recites:
wherein the second monitoring for detecting data drift based on a drop in accuracy of the first trained ML model includes using one or more of: Cumulative Sum (CUSUM) method, Adaptive Window (ADWIN) method, Early Drift Detection Method (EDDM), and Fast Hoeffding Drift Detection Method (FHDDM).
Elprin in view of Maughan teaches the method of claim 1 from which claim 7 depends, including the second monitoring for detecting data drift based on a drop in accuracy of the first trained ML model. Elprin in view of Maughan does not explicitly disclose includes one or more of: Cumulative Sum (CUSUM) method, Adaptive Window (ADWIN) method, Early Drift Detection Method (EDDM), and Fast Hoeffding Drift Detection Method (FHDDM).
However, Nasr-Azadani teaches in the field related to machine learning model production and model adaptation., Nasr-Azadani, para 1. Nasr-Azadani, which is analogous to the claimed invention because Nasr-Azadani is directed to concept drift and model adaptation, teaches that, As an example, FIG. 6 further illustrates a predictive accuracy drop 602 of a machine learning model as a function of time 604 due to concept drift. Such accuracy drop may be determined by comparing the prediction of the machine learning model in production and reality (e.g., whether a user has actually clicked a particular online advertisement or not in the example online streaming application above). The accuracy of the machine learning model is merely one of many metrics that may be used to represent the performance quality of the model. Time evolution of any other single or combinational metrics may be used as an indicator of the concept drift for the machine learning model. Nasr-Azadani, Figs 6, 7A-7F,8, para 26, 27-29, 3, 24, 40, 19-20.
The concept drift detection engine of FIGS. 8 and 9 may be based on single concept drive detector or an ensemble of concept drift detectors. The purpose of using a detector ensemble, for example, is to provide a flexibility to select multiple detectors of various characteristics that collectively provide more accurate detection of different types of concept drive for a particular application. The detection of concept drift may be based on comparison of predictions by the currently deployed production model and actual labels of the input streaming data. Each of the concept drift detectors may be based on various detection algorithms including but not limited to Drift Detection Method (DDM), Early Drift Detection Method (EDDM), Adaptive Windowing (ADWIN) detection, Page-Hinkely (P-H) statistical detection, and CUmulative SUM (CUSUM) detection (includes one or more of: Cumulative Sum (CUSUM) method, Adaptive Window (ADWIN) method, Early Drift Detection Method (EDDM), and Fast Hoeffding Drift Detection Method (FHDDM)). The detection output may be based on single or combinational metrics including but not limited to prediction accuracy, prediction precision, and detection delay. An ensemble detection approach, as described in more detail below, may facilitate increasing detection precision at the expense of, for example, increased detection delay due to added processing complexity. The detection engine performs a detection procedure to detect a concept drift of a machine learning model using the any combination of the approaches above. Nasr-Azadani, para 39, 26.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the model monitoring for handling data drift, selected drift adapter, performance metrics requirements, first trained ML model to handle the data drift of Elprin using the feature for testing the selected data drift adaptor to determine if the performance metrics requirements are met, and if the performance metrics requirements are met, applying the selected data drift adaptor to the first trained ML model to adapt the first trained ML model to handle the data drift of Maughan and using the second monitoring for detecting data drift based on a drop in accuracy of a first trained ML model that includes one or more of: Cumulative Sum (CUSUM) method, Adaptive Window (ADWIN) method, Early Drift Detection Method (EDDM), and Fast Hoeffding Drift Detection Method (FHDDM).of Nasr-Azadani, with a reasonable expectation of success, in order to provide for testing and selecting the most accurate model for continued use, because changes in data may cause a predictive model to become less accurate over time, even if the predictive model was initially accurate and in order to provide for adaptive adjustment of the machine learning model in production to mitigate the effect of predictive performance drop due to the concept drift. Maughan, para 5, 84. Nasr-Azadani, abstract, Figs 6, 7A-7F, para 2-3, 26-27, 39.
This would have provided the advantages of using an improved model with more accurate performance.
Allowable Subject Matter
Claims 3-4, 5-6 and 8-17 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the rejections under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite are overcome and if the rejections under 35 U.S.C. 101 as being directed to an abstract idea without significantly more are overcome.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US-20240152820-A1, US-20160071027-A1, US-10599957-B2, US-20220215297-A1, US-20220124002-A1, US-20220036201-A1, US-20200285997-A1, US-11315030-B2, US-6594620-B1, US-20200387836-A1, US-20180136617-A1, US-20200380417-A1, US-20200012900-A1, US-12165017-B1, US-20210034960-A1.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BARBARA LEVEL whose telephone number is (303)297-4748. The examiner can normally be reached Monday through Friday 8:00 AM - 5:00 PM MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BARBARA M LEVEL/Examiner, Art Unit 2142
Prosecution Insights