DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Title
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. Examiner believes that the title of the invention is imprecise. A descriptive title indicative of the invention will help in proper indexing, classifying, searching, etc. See MPEP 606.01. However, the title of the invention should be limited to 500 characters. Examiner suggests including the aspect(s) of the claims which Applicant believes to be novel or nonobvious over the prior art.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 1 is a method claim. Claim 13 is a system claim. Claim 19 is a CRM claim. Therefore, claims 1, 13, and 19 are directed to either a process, machine, manufacture or composition of matter.
With respect to Claim 1:
Step 2A Prong 1:
performing…a plurality of different drift checks for the input data point and for the trained model using training dataset profile information for the trained model, wherein the training dataset profile information for the trained model comprises information about a training dataset used to train and generate the trained model, and wherein performing the plurality of different drift checks comprises comparing the input data point to the training dataset profile information (mental process – user can manually perform drift checks using the listed information)
generating…a report comprising information identifying the plurality of different drift checks and a result generated from performing each different drift check of the plurality of different drift checks (mental process – user can manually generate a report)
generating an overall drift evaluation based on the result associated with each different drift check of the plurality of different drift checks (mental process – user can manually generate an overall drift evaluation)
Step 2A Prong 2: This judicial exception is not integrated into a practical application. Additional elements:
receiving…an input data point and model information identifying a trained model that is to be used to generate a prediction for the input data point (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g))
by a first computing device (mere instructions to apply the exception using a generic computer component)
outputting the report including the prediction and an indication of an impact on the prediction that is determined based on the overall drift evaluation (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g))
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception. Additional elements:
receiving…an input data point and model information identifying a trained model that is to be used to generate a prediction for the input data point (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer)
by a first computing device (mere instructions to apply the exception using a generic computer component)
outputting the report including the prediction and an indication of an impact on the prediction that is determined based on the overall drift evaluation (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer)
Conclusion: The claim is not patent eligible.
Claims 13 and 19 are rejected on the same grounds as claim 1.
Regarding Claims 4-10, 12, 16-18: These limitations, as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitations in the mind. That is, nothing in the claim limitation precludes the step from practically being performed in the mind.
For claims 4, 16: the limitation encompasses the user manually identifying a dataset and generating the dataset profile.
For claims 5, 17: the limitation encompasses the user manually using the recited data, comparing the data to the profile, and determining whether the input data point comprises a value for each column in the plurality of columns.
For claims 6. 18: the limitation encompasses the user manually using the recited data, and comparing a value in the particular column in the input data point to one or more metrics in the set of metrics.
For claim 7: the limitation encompasses the user manually using the specific metrics, and determining a value is higher or lower than some metrics.
For claim 8: the limitation encompasses the user manually using the specific metrics, and comparing a value to a metric.
For claim 9: the limitation encompasses the user manually using specific training data and profile information, and comparing a value to different categorical values.
For claim 10: the limitation encompasses the user manually using specific training data and profile information, and determining if the set of column types is same as the plurality of column types 11 indicated in the training dataset profile information.
For claim 12: the limitation encompasses the user manually using specific training data and profile information, and determining whether a unit of measure associated with a value in the particular column in the input data point is same as or different from the particular unit of measure.
These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible.
Regarding Claims 2-3, 11, 14-15, 20: The limitation, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. That is, other than the additional elements, nothing in the claim limitation precludes the step from practically being performed in the mind.
For claims 2, 14, 20: the limitation includes the additional element of receiving, communicating, and outputting data; a second computing device; and using the trained model. The generating a prediction limitation encompasses the user manually generating a prediction.
For claims 3, 15: the limitation includes the additional element of accessing data from memory.
For claim 11: the limitation includes the additional element of calling a function.
These judicial exceptions are not integrated into a practical application. The additional element(s) of a second computing device are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer component. The additional element(s) of using the trained model recite merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). The additional element(s) of receiving, communicating, and outputting data; accessing data from memory; and calling a function recite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of a second computing device amount to no more than mere instructions to apply the exception using a generic computer component or operation. Mere instructions to apply an exception using a generic computer component or operation cannot provide an inventive concept. The additional element(s) of using the trained model recite adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f). The additional element(s) of receiving, communicating, and outputting data; accessing data from memory; and calling a function recite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 11, 13-16, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (hereinafter Lu), Learning under Concept Drift: A Review, in view of Sallee et al. (hereinafter Sallee), U.S. Patent Application Publication 2023/0025677, further in view of Schierz et al. (hereinafter Schierz), U.S. Patent Application Publication 2021/0390455.
Regarding Claim 1, Lu discloses a computer implemented method, comprising:
receiving, by a first computing device, an input data point and model information identifying a trained model that is to be used to generate a prediction for the input data point [“Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2; “DDM needs a classifier to make the predictions. This process converts training data to a learning model, which is considered as the Stage 2 (Data Modeling).” §3.2.1 ¶2; Fig. 2];
performing, by the first computing device, a plurality of different drift checks for the input data point and for the trained model using training dataset profile information for the trained model, [“Concept drift occurs at timestamp t+1, if F0,t(X,y) ≠ Ft+1,∞ (X, y), denoted as ⱻt:Pt(X, y) ≠ Pt+1(X, y).” §2.1 ¶1], wherein the training dataset profile information for the trained model comprises information about a training dataset used to train and generate the trained model [“a set of samples, denoted as S0,t” §2.1 ¶1; S0,t follows a certain distribution F0,t (X, y)” §2.1 ¶1], and wherein performing the plurality of different drift checks comprises comparing the input data point to the training dataset profile information [“Concept drift occurs at timestamp t+1, if F0,t(X,y) ≠ Ft+1,∞ (X, y), denoted as ⱻt:Pt(X, y) ≠ Pt+1(X, y).” §2.1 ¶1]; and
generating, by the first computing device, a report comprising information identifying the plurality of different drift checks and a result generated from performing each different drift check of the plurality of different drift checks [“These algorithms address concept drift from the root sources, which is the distribution drift. Not only can they accurately identify the time of drift, they can also provide location information about the drift.” §3.2.2 ¶1; “This status information is the output of the drift detection algorithms, and is used as input for drift adaptation.” §4 ¶1];
outputting the report [“This status information is the output of the drift detection algorithms” §4 ¶1] including the prediction [“Prediction” Fig. 2] and an indication of an impact on the prediction that is determined based on the overall drift evaluation.
However, Lu fails to explicitly disclose by a first computing device.
Sallee discloses by a first computing device [“The one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet)” ¶51].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu and Sallee before him before the effective filing date of the claimed invention, to modify the method of Lu to incorporate distributed computing of Sallee.
Given the advantage of faster computing by using various computing systems for parts of the operation, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Lu and Sallee fail to explicitly disclose a plurality of different drift checks;
generating an overall drift evaluation based on the result associated with each different drift check of the plurality of different drift checks; and
outputting the report including the prediction and an indication of an impact on the prediction that is determined based on the overall drift evaluation.
Schierz discloses a plurality of different drift checks [“Each feature in the data can be assessed individually using the adaptive drift learner 126, which can predict ( e.g., using artificial intelligence) a best or preferred binning strategy and drift metric to use for that feature and/or can apply anomaly detection to detect any changes or drift in the data. To make these predictions, the adaptive drift learner 126 can consider various factors related to the feature, such as whether the feature is represented by or includes numeric data, category data, or text data. Additionally or alternatively, the adaptive drift learner 126 can consider: whether numerical values are integers or floating point values; how many levels or categories are included in category data; whether a feature is seasonal and, if so, whether drift is expected; and/or how much data is missing for the feature.” ¶115];
generating an overall drift evaluation based on the result associated with each different drift check of the plurality of different drift checks [Each feature in the data can be assessed individually using the adaptive drift learner 126, which can predict ( e.g., using artificial intelligence) a best or preferred binning strategy and drift metric to use for that feature and/or can apply anomaly detection to detect any changes or drift in the data. To make these predictions, the adaptive drift learner 126 can consider various factors related to the feature, such as whether the feature is represented by or includes numeric data, category data, or text data. Additionally or alternatively, the adaptive drift learner 126 can consider: whether numerical values are integers or floating point values; how many levels or categories are included in category data; whether a feature is seasonal and, if so, whether drift is expected; and/or how much data is missing for the feature.” ¶115]; and
outputting the report including the prediction and an indication of an impact on the prediction that is determined based on the overall drift evaluation [Figs. 4B, 5-9].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, and Schierz before him before the effective filing date of the claimed invention, to modify the combination to incorporate the different drift checks, overall evaluation, and outputting an impact of the drift of Schierz.
Given the advantage of more accurate drift analysis, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 2, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses further comprising:
receiving, …, the input data point [“Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2; Fig. 2];
generating, …, the prediction for the input data point using the trained model [“a classifier to make the predictions” §3.2.1 ¶2]; and
along with the report [“This status information is the output of the drift detection algorithms” §4 ¶1].
However, Lu fails to explicitly disclose by a second computing device; wherein the outputting the report comprises communicating the report from the first computing device to the second computing device; and
outputting, by the second computing device, the prediction.
Sallee discloses by a second computing device; wherein the outputting the report comprises communicating the report from the first computing device to the second computing device [“The one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet)” ¶51]; and
outputting, by the second computing device, the prediction [“The monitor 106 can receive output of the ML model 110” ¶11].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, and Schierz before him before the effective filing date of the claimed invention, to modify the combination to incorporate distributed computing of Sallee.
Given the advantage of faster computing by using various computing systems for parts of the operation, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 3, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses further comprising:
accessing…and based upon the model information, the training dataset profile information [“Concept drift occurs at timestamp t+1, if F0,t(X,y) ≠ Ft+1,∞ (X, y), denoted as ⱻt:Pt(X, y) ≠ Pt+1(X, y).” §2.1 ¶1].
However, Lu fails to explicitly disclose by the first computing device and from a memory location.
Sallee discloses by the first computing device and from a memory location [“The one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet)” ¶51].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, and Schierz before him before the effective filing date of the claimed invention, to modify the combination to incorporate distributed computing of Sallee.
Given the advantage of faster computing by using various computing systems for parts of the operation, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 4, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses further comprising:
identifying, …, the training dataset used to train and generate the trained model [“Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2]; and
generating, …, at least a portion of the training dataset profile information based upon the training dataset [“Concept drift occurs at timestamp t+1, if F0,t(X,y) ≠ Ft+1,∞ (X, y), denoted as ⱻt:Pt(X, y) ≠ Pt+1(X, y).” §2.1 ¶1].
However, Lu fails to explicitly disclose by the first computing device.
Sallee discloses by the first computing device [“The one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet)” ¶51].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, and Schierz before him before the effective filing date of the claimed invention, to modify the combination to incorporate distributed computing of Sallee.
Given the advantage of faster computing by using various computing systems for parts of the operation, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 11, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1.
However, Lu fails to explicitly disclose for at least one different drift check in the plurality of different drift checks, calling a serverless function to perform the at least one different drift check.
Sallee discloses for at least one different drift check in the plurality of different drift checks, calling a serverless function to perform the at least one different drift check [“at least some of the operations of a method may be performed by one or processors or processor-implemented module” ¶50; “the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment” ¶57].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, and Schierz before him before the effective filing date of the claimed invention, to modify the combination to incorporate the single processor standalone computer or client-side operation of Sallee.
Given the advantage of faster processing by removing communication time or server wait time in a high traffic system, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Lu fails to explicitly disclose wherein performing the plurality of different drift checks comprises:
for at least one different drift check in the plurality of different drift checks, calling a serverless function to perform the at least one different drift check.
Schierz discloses wherein performing the plurality of different drift checks comprises:
for at least one different drift check in the plurality of different drift checks, calling a serverless function to perform the at least one different drift check [“Each feature in the data can be assessed individually using the adaptive drift learner 126, which can predict ( e.g., using artificial intelligence) a best or preferred binning strategy and drift metric to use for that feature and/or can apply anomaly detection to detect any changes or drift in the data. To make these predictions, the adaptive drift learner 126 can consider various factors related to the feature, such as whether the feature is represented by or includes numeric data, category data, or text data. Additionally or alternatively, the adaptive drift learner 126 can consider: whether numerical values are integers or floating point values; how many levels or categories are included in category data; whether a feature is seasonal and, if so, whether drift is expected; and/or how much data is missing for the feature.” ¶115].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, and Schierz before him before the effective filing date of the claimed invention, to modify the combination to incorporate the different drift checks of Schierz.
Given the advantage of more accurate drift analysis, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claims 13 and 19 are rejected on the same grounds as claim 1.
Claims 14-16 are rejected on the same grounds as claims 2-4, respectively.
Claim 20 is rejected on the same grounds as claim 2.
Claim(s) 5-10, 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu, Sallee, and Schierz, further in view of Demsar et al. (hereinafter Demsar), Detecting concept drift in data streams using model explanation.
Regarding Claim 5, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses wherein:
the training dataset comprises a plurality of training input data points [“a set of samples” §2.1 ¶2; “Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2], each training input data point in the plurality of training input data points comprising a plurality of columns [“denoted as S0,t={d0,…,dt}, where di = (Xi, yi) is one observation (or data instance), Xi is the feature vector” §2.1 ¶2; Note: a vector has a plurality of columns of size 1];
the training dataset profile information comprises information identifying the plurality of columns [“Xi is the feature vector” and “F0,t(X,y)” §2.1 ¶2].
However, Lu fails to explicitly disclose comparing the input data point to the training dataset profile information comprises determining whether the input data point comprises a value for each column in the plurality of columns.
Demsar discloses comparing the input data point to the training dataset profile information comprises determining whether the input data point comprises a value for each column in the plurality of columns [“monitors the explanation (i.e., contribution) of individual attribute values” §3.3 ¶1; Note: monitoring values determines the value of the data point].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, and Demsar before him before the effective filing date of the claimed invention, to modify the combination to incorporate individual value comparisons over time of Demsar.
Given the advantage of fine granular review of data drift to catch model drift detection to ensure accurate models, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 6, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses wherein:
the training dataset comprises a plurality of training input data points [“a set of samples” §2.1 ¶2; “Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2], each training input data point in the plurality of training input data points comprising a plurality of columns [“denoted as S0,t={d0,…,dt}, where di = (Xi, yi) is one observation (or data instance), Xi is the feature vector” §2.1 ¶2; Note: a vector has a plurality of columns of size 1].
However, Lu fails to explicitly disclose the training dataset profile information comprises, for a first column in the plurality of columns, information identifying a set of metrics determined based upon numerical values in the first column for the plurality of training input data points; and
comparing the input data point to the training dataset profile information comprises: for a particular column in the input data point corresponding to the first column, comparing a value in the particular column in the input data point to one or more metrics in the set of metrics.
Demsar discloses the training dataset profile information comprises, for a first column in the plurality of columns, information identifying a set of metrics determined based upon numerical values in the first column for the plurality of training input data points [“Calculate average dissimilarity to other explanations” Algorithm 1; “monitors the explanation (i.e., contribution) of individual attribute values (if the attributes are continuous, the IME discretizes them into intervals)” §3.3 ¶1]; and
comparing the input data point to the training dataset profile information comprises: for a particular column in the input data point corresponding to the first column, comparing a value in the particular column in the input data point to one or more metrics in the set of metrics [“monitors the explanation (i.e., contribution) of individual attribute values” §3.3 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, and Demsar before him before the effective filing date of the claimed invention, to modify the combination to incorporate individual value comparisons of Demsar.
Given the advantage of fine granular review of data drift to catch model drift detection to ensure accurate models, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 7, Lu, Sallee, Schierz, and Demsar disclose the computer implemented method of claim 6.
However, Lu fails to explicitly disclose wherein:
the set of metrics includes a first metric indicative of a lowest numerical value in the first column in the training dataset and a second metric indicative of a highest numerical value in the first column in the training dataset;
comparing the value in the particular column in the input data point to one or more metrics in the set of metrics comprises determining whether the value in the particular column is lower than the first metric and higher than the second metric.
Demsar discloses wherein:
the set of metrics includes a first metric indicative of a lowest numerical value in the first column in the training dataset and a second metric indicative of a highest numerical value in the first column in the training dataset [“Euclidean distance as the dissimilarity measure d, and set the explanation granularity to 50.” §4 ¶3; “drift widths” §4.2 ¶1];
comparing the value in the particular column in the input data point to one or more metrics in the set of metrics comprises determining whether the value in the particular column is lower than the first metric and higher than the second metric [“Euclidean distance as the dissimilarity measure d, and set the explanation granularity to 50.” §4 ¶3; “drift widths” §4.2 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, and Demsar before him before the effective filing date of the claimed invention, to modify the combination to incorporate deviation ranges for values of Demsar.
Given the advantage of setting a proper deviation range to ensure accurate detection of drift, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 8, Lu, Sallee, Schierz, and Demsar disclose the computer implemented method of claim 6.
However, Lu fails to explicitly disclose wherein:
the set of metrics includes a first metric indicative of a mean value based upon numerical values in the first column in the training dataset;
comparing the value in the particular column in the input data point to one or more metrics in the set of metrics comprises comparing the value in the particular column to the first metric.
Demsar discloses wherein:
the set of metrics includes a first metric indicative of a mean value based upon numerical values in the first column in the training dataset [“their mean is then proclaimed to be the explanation of feature value a” §2.3 ¶5; “mean positive and negative contribution of the attribute” §5.1 ¶2];
comparing the value in the particular column in the input data point to one or more metrics in the set of metrics comprises comparing the value in the particular column to the first metric [“monitors the explanation (i.e., contribution) of individual attribute values” §3.3 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, and Demsar before him before the effective filing date of the claimed invention, to modify the combination to incorporate mean used in the determination of values of Demsar.
Given the advantage of detecting drift using averages to allow for variation without constantly concluding drift has occurred, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 9, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses wherein:
the training dataset comprises a plurality of training input data points [“a set of samples” §2.1 ¶2; “Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2], each training input data point in the plurality of training input data points comprising a plurality of columns [“denoted as S0,t={d0,…,dt}, where di = (Xi, yi) is one observation (or data instance), Xi is the feature vector” §2.1 ¶2; Note: a vector has a plurality of columns of size 1].
However, Lu fails to explicitly disclose the training dataset profile information comprises, for a first column in the plurality of columns, information identifying a set of different categorical values in the first column for the plurality of training input data points;
comparing the input data point to the training dataset profile information comprises: for a particular column in the input data point corresponding to the first column, comparing a value in the particular column in the input data point to the set of different categorical values.
Demsar discloses the training dataset profile information comprises, for a first column in the plurality of columns, information identifying a set of different categorical values in the first column for the plurality of training input data points [“categorical attributes with a large number of distinct values” §4.6 ¶4];
comparing the input data point to the training dataset profile information comprises: for a particular column in the input data point corresponding to the first column, comparing a value in the particular column in the input data point to the set of different categorical values [“monitors the explanation (i.e., contribution) of individual attribute values” §3.3 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, and Demsar before him before the effective filing date of the claimed invention, to modify the combination to incorporate categorical analysis of Demsar.
Given the advantage of easy interpretability of values for drift detection using categorical data, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 10, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses wherein:
the training dataset comprises a plurality of training input data points [“a set of samples” §2.1 ¶2; “Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2], each training input data point in the plurality of training input data points comprising a plurality of columns, the plurality of columns corresponding to a plurality of column types [“denoted as S0,t={d0,…,dt}, where di = (Xi, yi) is one observation (or data instance), Xi is the feature vector” §2.1 ¶2; Note: a vector has a plurality of columns of size 1];
the training dataset profile information comprises information indicative of the plurality of column types [Xi is the feature vector” §2.1 ¶2].
However, Lu fails to explicitly disclose comparing the input data point to the training dataset profile information comprises: for a set of column types corresponding to a set of columns in the input data point, determining if the set of column types is same as the plurality of column types indicated in the training dataset profile information.
Demsar discloses comparing the input data point to the training dataset profile information comprises: for a set of column types corresponding to a set of columns in the input data point, determining if the set of column types is same as the plurality of column types indicated in the training dataset profile information [“monitors the explanation (i.e., contribution) of individual attribute values” §3.3 ¶1; Note: Monitoring values includes value types such as those seen in Fig 1; “ODOR” “GILL-SPACING” “VEIL-COLOR” “RING-NUMBER” “GILL-SIZE” “STALK-ROOT” Fig 1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, and Demsar before him before the effective filing date of the claimed invention, to modify the combination to incorporate different feature types of Demsar.
Given the advantage of comparing like types of values to each other to get meaningful information, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claims 17-18 are rejected on the same grounds as claims 5-6, respectively.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lu, Sallee, and Schierz, further in view of Demsar, further in view of Herring, Data Drift in Azure Machine Learning.
Regarding Claim 12, Lu, Sallee, and Schierz disclose the computer implemented method of claim 1. Lu further discloses wherein:
the training dataset comprises a plurality of training input data points [“a set of samples” §2.1 ¶2; “Stage 1 (Data Retrieval) aims to retrieve data chunks from data streams.” §3.1 ¶2], each training input data point in the plurality of training input data points comprising a plurality of columns [“denoted as S0,t={d0,…,dt}, where di = (Xi, yi) is one observation (or data instance), Xi is the feature vector” §2.1 ¶2; Note: a vector has a plurality of columns of size 1].
However, Lu fails to explicitly disclose the training dataset profile information comprises, for a first column in the plurality of columns,
comparing the input data point to the training dataset profile information comprises: for a particular column in the input data point corresponding to the first column.
Demsar discloses the training dataset profile information comprises, for a first column in the plurality of columns [“monitors the explanation (i.e., contribution) of individual attribute values” §3.3 ¶1],
comparing the input data point to the training dataset profile information comprises: for a particular column in the input data point corresponding to the first column [“monitors the explanation (i.e., contribution) of individual attribute values” §3.3 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, and Demsar before him before the effective filing date of the claimed invention, to modify the combination to incorporate the individual attribute evaluation of Demsar.
Given the advantage of fine granular review of data drift to catch model drift detection to ensure accurate models, one having ordinary skill in the art would have been motivated to make this obvious modification.
However, Lu fails to explicitly disclose information identifying a particular unit of measure associated with values in the first column in the plurality of training input data points; and
determining whether a unit of measure associated with a value in the particular column in the input data point is same as or different from the particular unit of measure.
Herring discloses information identifying a particular unit of measure associated with values in the first column in the plurality of training input data points [Causes of data drift include: • Upstream process changes, such as a sensor being replaced that change the unit of measure being applied. Maybe from Imperial to Metric?” pg. 1]; and
determining whether a unit of measure associated with a value in the particular column in the input data point is same as or different from the particular unit of measure points [Causes of data drift include: • Upstream process changes, such as a sensor being replaced that change the unit of measure being applied. Maybe from Imperial to Metric?” pg. 1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Lu, Sallee, Schierz, Demsar, and Herring before him before the effective filing date of the claimed invention, to modify the combination to incorporate the unit of measure difference detection for data drift.
Given the advantage of increased accuracy by determining that data drift is caused from a change in the unit of measure, one having ordinary skill in the art would have been motivated to make this obvious modification.
Examiner’s Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification.
Response to Arguments
Regarding the §101 rejections, Applicant's arguments have been fully considered but have been found unpersuasive. Applicant argues that the claimed invention provides a practical application and is thus patent eligible. Examiner disagrees for at least the following reasons.
As outlined in the rejections above, performing drift checks, generating a report, and generating an overall drift evaluation are abstract elements that can be completed mentally or with pen and paper. The additional elements of implementation on a computing device, receiving data, and outputting data are additional elements that do not integrate the abstract idea into a practical application. Specifically, receiving and outputting data are merely adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g), and the use of a computing device is merely instructions to apply the exception using a generic computer component. Examiner is not persuaded by Applicant’s assertion that the claimed invention provides a framework that is inextricably tied to computer technology. Applicant even admits in their Remarks (pg. 12) that detecting drift is typically performed by data scientists. Accordingly, the abstract elements can be performed manually by a person. Since the additional elements do not integrate the abstract idea into a practical application, for at least these reasons, the rejections are maintained.
Regarding the prior art rejections, Applicant's arguments with respect to the claims have been considered but are moot because the arguments do not apply to the references being used in the current rejection of the limitations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.B./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148