DETAILED ACTION
This communication is in response to the Application No. 18/365,364 filed July 29, 2026
in which Claims 1-7, and 9-21 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The amendments/arguments filed on July 29, 2026 have been considered. Claims 1-3, 6-7, and 9-20 have been amended. Claim 8 has been cancelled. Claim 21 has been added. Thus, claims 1-7, and 9-21 are pending and presented for examination.
Applicant’s arguments filled on July 29, 2026 with respect to the 35 U.S.C. 101 rejection have been fully considered and they are not persuasive.
Response to Applicant’s arguments on pgs. 10-11:
Examiner respectfully disagrees. Applicant argues that claim 1 "recites a specific, storage-efficient architecture for machine-learning performance monitoring that is neither abstract nor capable of being performed mentally," and that the human mind cannot practically "generate a performance output based on a comparison between the synthetic dataset and a contemporary input dataset". The limitation “generating […] a performance output for the machine learning model based on a comparison between the synthetic dataset and a contemporary input dataset, wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset" is recited at a high level of generality and may be practically performed by mental process. For example, a user may observe/analyze the synthetic dataset and the contemporary input dataset, compare the values or distributions of the two datasets, and accordingly use judgement/evaluation to determine whether the datasets differ in a manner indicating a data drift abnormality, then note that determination as a performance output, mentally or with pen and paper. Similarly, the limitation "identifying […] a generative synthetic data model corresponding to a historical training dataset previously used to train a machine learning model" may be practically performed by mental process, as a user may observe/analyze the available models and accordingly use judgement/evaluation to select the model that corresponds to the training dataset. The claim does not limit the size or nature of either dataset and does not recite any particular comparison technique. Thus, contrary to Applicant's reliance on the MPEP's statement that claims do not recite a mental process "when the human mind is not equipped to perform the claim limitation", the human mind is equipped to compare two sets of data and judge whether they differ. The limitation falls within the mental processes grouping.
Applicant further argues that the human mind cannot practically “store a generative synthetic data model in place of a historical training dataset, remove the historical training dataset from computer storage while preserving its functionality through the stored model, or generate a synthetic dataset using the generative synthetic data model,” and that the claimed process involves “storing, removing, regenerating, and comparing datasets at scale”. This argument treats the additional elements as if they were the abstract idea itself, but they are separate. The Examiner does not contend that the “wherein (i) the generative synthetic data model is stored in place of the historical training dataset […] and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored” limitation, or the “generating, […] using the generative synthetic data model, a synthetic dataset” step are mental processes. Those limitations are additional elements, addressed at Step 2A Prong 2 and Step 2B. Even where certain limitations cannot be performed mentally, a claim that includes a limitation reciting a mental process still recites a judicial exception. In addition, the claims do not recite any particular "scale." Performing an evaluation faster or on a larger volume of data with a computer does not make it any less an abstract idea when a person could otherwise carry it out. Claim 1 also does not recite that the dataset's “functionality” is preserved “through the stored model” the claim requires only that the dataset be “removed from storage.”
Response to Applicant’s arguments on pgs. 11-13:
Examiner respectfully disagrees. Applicant argues that the limitation “wherein (i) the generative synthetic data model is stored in place of the historical training dataset […] and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored” reduces storage overhead and improves computer functionality. An improvement described in the specification must be reflected in the claim. The quoted limitation, however, does not require the model to be “compact,” to occupy less storage than the dataset, or to be stored in any particular manner. It merely replaces one stored item with another. For example, a large generative network trained on a modest dataset may occupy more storage than the dataset it replaces, yet such an arrangement falls squarely within the claim. Because the claim covers embodiments in which no storage reduction results, it recites the idea of a solution rather than a particular way of achieving the asserted improvement.
Applicant also emphasizes the limitation “generating […] a performance output […] based on a comparison between the synthetic dataset and a contemporary input dataset”. That limitation is the abstract idea itself, and an improvement in the abstract idea is not an improvement in technology. The claimed comparison evaluates drift in the same way whether the reference data is the original dataset or a synthetic stand-in for it. The claim changes only where the reference data comes from, not how the comparison is performed or how the computer operates. Moreover, the features Applicant relies on to establish a storage benefit, namely a “condensed, serializable representation within a composite model data object” and a model that “preserves the dataset's evaluative utility”, are not recited in claim 1. Reduced governance burden and data-exposure risk are administrative benefits of retaining less information rather than technological improvements.
Regarding Ex parte Desjardins argument, claim 1 recites only outcomes. Neither the monitored model nor the generative model operates any differently as a result of the claimed steps. The additional elements, considered individually and in combination, do not integrate the exception into a practical application or amount to significantly more. The processors are generic computer components, and “generating, […] using the generative synthetic data model, a synthetic dataset” invokes a generic model as a tool. See MPEP 2106.05(f). Likewise, “initiating […] a model performance-based operation” merely acts on the result of the abstract idea. Storing data is also well-understood, routine, and conventional. See MPEP 2106.05(d)(II). Claims 13 and 18 recite substantially the same limitations on generic computer components and are not patent eligible for the same reasons.
Thus, the 35 U.S.C. 101 rejection is maintained.
Applicant’s arguments filed on July 29, 2026 with respect to the 35 U.S.C. 103 rejection have been fully considered but are moot because of the new ground of rejection.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-7 and 9-21 are rejected under 35 U.S.C. 101 because these claimed inventions are directed to an abstract idea without significantly more.
Regarding Claim 1:
Step 1: Claim 1 is a method type claim. Therefore, Claims 1-12 fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance
of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable
interpretation, covers performance of the limitation by mathematical calculation but for the recitation
of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract
ideas.
Identifying, […], a generative synthetic data model corresponding to a historical training dataset […](mental process – identifying a generative synthetic data model may be performed mentally a user observing/analyzing the historical training dataset and accordingly using judgement/evaluation to identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model based on said analysis)
generating […] a synthetic dataset for the machine learning model (mental process – generating a synthetic dataset may be performed mentally by a user observing/analyzing the machine learning model and accordingly using judgement/evaluation to generate a synthetic dataset for the machine learning model based on said analysis)
Generating […] a performance output for the machine learning model based on a comparison between the synthetic dataset and a contemporary input dataset (mental process – generating a performance output for the machine learning model may be performed mentally by a user observing/analyzing the synthetic dataset and the contemporary input dataset and accordingly using judgement/evaluation to generate a performance output for the machine learning model based on said analysis)
and initiating […] performance of a model performance-based operation based on the performance output for the machine learning model (mental process – initiating the performance of one or more model performance-based operations may be performed mentally by a user observing/analyzing the performance output for the machine learning model and using judgement/evaluation to decide whether to initiate a responsive operation based on said analysis. For example, as stated in the specification (Par. [0067]), a model performance-based operation may include initiating the performance of one or more alerts, messages, instructions, and/or the like in response to a performance output.)
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] previously used to train a machine learning model […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model without significantly more)
wherein (i) the generative synthetic data model is stored in place of the historical training dataset for use in evaluating post-deployment performance of the machine learning model and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored (Adding insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g).)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset(Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
Step 2B: The claim does not include additional elements considered individually and in combination that
are sufficient to amount to significantly more than the judicial exception.
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] previously used to train a machine learning model […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model without significantly more)
wherein (i) the generative synthetic data model is stored in place of the historical training dataset for use in evaluating post-deployment performance of the machine learning model and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored (MPEP 2106.05(d)(II) indicates that "Storing and retrieving information in memory" and “Electronic recordkeeping” are well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim))
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset(Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 1 – 7 and 9-12. The additional limitations of the dependent claims are addressed below.
Regarding Claim 2:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 2 depends on.
Step 2A Prong 2 & Step 2B:
wherein a data model representation corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that a data model representation corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 3:
Step 2A Prong 1: See the rejection of Claim 2 above, which Claim 3 depends on.
Step 2A Prong 2 & Step 2B:
wherein the model registry comprises a plurality of composite model data objects and a composite model data object of the plurality of composite model data objects comprises a respective machine learning model and a respective data model representation corresponding a respective generative synthetic data model corresponding to the respective machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the model registry comprises a plurality of composite model data objects and a composite model data object of the plurality of composite model data objects comprises a respective machine learning model and a respective data model representation corresponding a respective generative synthetic data model corresponding to the respective machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 2. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 4:
Step 2A Prong 1: See the rejection of Claim 2 above, which Claim 4 depends on.
Step 2A Prong 2 & Step 2B:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the data model representation comprising a serialized representation of the generative synthetic data model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 2. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 2A Prong 1: See the rejection of Claim 4 above, which Claim 5 depends on.
generating the generative synthetic data model by deserializing the data model representation(mental process – generating the generative synthetic data model may be performed mentally by a user observing/analyzing the data model representation and using judgment/evaluation to reconstruct a model from that representation based on said analysis. For example, the specification explains that a generative synthetic data model may include a statistical model, such as a linear regression model or tree-based model, and a user can formulate a simple statistical model using pen and paper (Par. [0055]))
And generating […] a plurality of evaluation samples corresponding to the historical training dataset (mental process – generating a plurality of evaluation samples may be performed mentally by a user observing/analyzing the historical training data and accordingly using judgment/evaluation to generate a plurality of evaluation samples corresponding to the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
[…], using the generative synthetic data model, […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 4. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 6:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 6 depends on.
[…]to generate the synthetic dataset representing the historical training dataset (mental process – generate synthetic datasets may be performed mentally by a user observing/analyzing the historical training dataset and using judgment/evaluation to generate one or more synthetic datasets representing the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
wherein the generative synthetic data model comprises a second machine learning model previously trained using the historical training dataset […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a trained machine learning model without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 7:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 7 depends on.
Step 2A Prong 2 & Step 2B:
wherein the generative synthetic data model is previously generated within a training time period corresponding to a training operation for the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the generative synthetic data model is previously generated within a training time period corresponding to a training operation for the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 9:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 9 depends on.
Step 2A Prong 2 & Step 2B:
wherein the performance output is generated for the machine learning model at a data drift monitoring frequency (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output is generated for the machine learning model at a data drift monitoring frequency does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 10:
Step 2A Prong 1: See the rejection of Claim 9 above, which Claim 10 depends on.
Step 2A Prong 2 & Step 2B:
wherein the data drift monitoring frequency defines an evaluation time period and the contemporary input dataset comprises a plurality of input data objects corresponding to the evaluation time period (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the data drift monitoring frequency defines an evaluation time period and the contemporary input dataset comprising a plurality of input data objects corresponding to the evaluation time period does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 9. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 11:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 11 depends on.
Step 2A Prong 2 & Step 2B:
retraining the machine learning model in response to the performance output failing to satisfy a performance threshold for the machine learning model (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of model retraining without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 12:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 12 depends on.
Step 2A Prong 2 & Step 2B:
wherein the model performance-based operation is initiated based on a comparison between the performance output and a performance threshold for the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the model performance-based operation is initiated based on a comparison between the performance output and a performance threshold does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 13:
Step 1: Claim 13 is a system type claim. Therefore, Claims 13-17 and 21 fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance
of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable
interpretation, covers performance of the limitation by mathematical calculation but for the recitation
of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract
ideas.
Identifying, […], a generative synthetic data model corresponding to a historical training dataset […](mental process – identifying a generative synthetic data model may be performed mentally a user observing/analyzing the historical training dataset and accordingly using judgement/evaluation to identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model based on said analysis)
generating […] a synthetic dataset for the machine learning model (mental process – generating a synthetic dataset may be performed mentally by a user observing/analyzing the machine learning model and accordingly using judgement/evaluation to generate a synthetic dataset for the machine learning model based on said analysis)
Generating […] a performance output for the machine learning model based on a comparison between the synthetic dataset and a contemporary input dataset (mental process – generating a performance output for the machine learning model may be performed mentally by a user observing/analyzing the synthetic dataset and the contemporary input dataset and accordingly using judgement/evaluation to generate a performance output for the machine learning model based on said analysis)
and initiating […] performance of a model performance-based operation based on the performance output for the machine learning model (mental process – initiating the performance of one or more model performance-based operations may be performed mentally by a user observing/analyzing the performance output for the machine learning model and using judgement/evaluation to decide whether to initiate a responsive operation based on said analysis. For example, as stated in the specification (Par. [0067]), a model performance-based operation may include initiating the performance of one or more alerts, messages, instructions, and/or the like in response to a performance output.)
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
[…]memory and one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] previously used to train a machine learning model […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model without significantly more)
wherein (i) the generative synthetic data model is stored in place of the historical training dataset for use in evaluating post-deployment performance of the machine learning model and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored (Adding insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g).)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…] wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset(Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Step 2B: The claim does not include additional elements considered individually and in combination that
are sufficient to amount to significantly more than the judicial exception.
[…]memory and one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] previously used to train a machine learning model […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model without significantly more)
wherein (i) the generative synthetic data model is stored in place of the historical training dataset for use in evaluating post-deployment performance of the machine learning model and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored (MPEP 2106.05(d)(II) indicates that "Storing and retrieving information in memory" and “Electronic recordkeeping” are well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim))
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…] wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset(Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
For the reasons above, Claim 13 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 13 – 17 and 21. The additional limitations of the dependent claims are addressed below.
Regarding Claim 14:
Step 2A Prong 1: See the rejection of Claim 13 above, which Claim 14 depends on.
Step 2A Prong 2 & Step 2B:
wherein a data model representation corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that a data model representation corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 13. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 15:
Step 2A Prong 1: See the rejection of Claim 14 above, which Claim 15 depends on.
Step 2A Prong 2 & Step 2B:
wherein the model registry comprises a plurality of composite model data objects and a composite model data object of the plurality of composite model data objects comprises a respective machine learning model and a respective data model representation corresponding a respective generative synthetic data model corresponding to the respective machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the model registry comprises a plurality of composite model data objects and a composite model data object of the plurality of composite model data objects comprises a respective machine learning model and a respective data model representation corresponding a respective generative synthetic data model corresponding to the respective machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 14. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 16:
Step 2A Prong 1: See the rejection of Claim 14 above, which Claim 16 depends on.
Step 2A Prong 2 & Step 2B:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the data model representation comprising a serialized representation of the generative synthetic data model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 14. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 17:
Step 2A Prong 1: See the rejection of Claim 16 above, which Claim 17 depends on.
generating the generative synthetic data model by deserializing the data model representation (mental process – generating the generative synthetic data model may be performed mentally by a user observing/analyzing the data model representation and using judgment/evaluation to reconstruct a model from that representation based on said analysis. For example, the specification explains that a generative synthetic data model may include a statistical model, such as a linear regression model or tree-based model, and a user can formulate a simple statistical model using pen and paper (Par. [0055]))
And generating […] a plurality of evaluation samples corresponding to the historical training dataset (mental process – generating a plurality of evaluation samples may be performed mentally by a user observing/analyzing the historical training data and accordingly using judgment/evaluation to generate a plurality of evaluation samples corresponding to the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
[…], using the generative synthetic data model, […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 16. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 18:
Step 1: Claim 18 is a non-transitory computer-readable storage media type claim. Therefore, Claims 18-20 fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance
of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable
interpretation, covers performance of the limitation by mathematical calculation but for the recitation
of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract
ideas.
Identifying, […], a generative synthetic data model corresponding to a historical training dataset […](mental process – identifying a generative synthetic data model may be performed mentally a user observing/analyzing the historical training dataset and accordingly using judgement/evaluation to identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model based on said analysis)
generating […] a synthetic dataset for the machine learning model (mental process – generating a synthetic dataset may be performed mentally by a user observing/analyzing the machine learning model and accordingly using judgement/evaluation to generate a synthetic dataset for the machine learning model based on said analysis)
Generating […] a performance output for the machine learning model based on a comparison between the synthetic dataset and a contemporary input dataset (mental process – generating a performance output for the machine learning model may be performed mentally by a user observing/analyzing the synthetic dataset and the contemporary input dataset and accordingly using judgement/evaluation to generate a performance output for the machine learning model based on said analysis)
and initiating […] performance of a model performance-based operation based on the performance output for the machine learning model (mental process – initiating the performance of one or more model performance-based operations may be performed mentally by a user observing/analyzing the performance output for the machine learning model and using judgement/evaluation to decide whether to initiate a responsive operation based on said analysis. For example, as stated in the specification (Par. [0067]), a model performance-based operation may include initiating the performance of one or more alerts, messages, instructions, and/or the like in response to a performance output.)
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
[…]one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] previously used to train a machine learning model […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model without significantly more)
wherein (i) the generative synthetic data model is stored in place of the historical training dataset for use in evaluating post-deployment performance of the machine learning model and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored (Adding insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g).)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…] wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset(Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Step 2B: The claim does not include additional elements considered individually and in combination that
are sufficient to amount to significantly more than the judicial exception.
[…]memory and one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] previously used to train a machine learning model […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of training a machine learning model without significantly more)
wherein (i) the generative synthetic data model is stored in place of the historical training dataset for use in evaluating post-deployment performance of the machine learning model and (ii) the historical training dataset is removed from storage after the generative synthetic data model is stored (MPEP 2106.05(d)(II) indicates that "Storing and retrieving information in memory" and “Electronic recordkeeping” are well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim))
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…] wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset(Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset represented by the synthetic dataset does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
For the reasons above, Claim 18 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 18 - 20. The additional limitations of the dependent claims are addressed below.
Regarding Claim 19:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 19 depends on.
[…]to generate the synthetic dataset representing the historical training dataset (mental process – generate the synthetic datasets may be performed mentally by a user observing/analyzing the historical training dataset and using judgment/evaluation to generate one or more synthetic datasets representing the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
wherein the generative synthetic data model comprises a second machine learning model previously trained using the historical training dataset […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a trained machine learning model without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 18. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 20:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 20 depends on.
Step 2A Prong 2 & Step 2B:
wherein the generative synthetic data model is previously generated within a training time period corresponding to a training operation for the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 18. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 21:
Step 2A Prong 1: See the rejection of Claim 13 above, which Claim 21 depends on.
Step 2A Prong 2 & Step 2B:
retraining the machine learning model in response to the performance output failing to satisfy a performance threshold for the machine learning model (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of model retraining without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 13. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-12, 13, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Walters et al. (hereinafter Walters) (US 10599957), in view of Fly et al. (hereinafter Fly) (US 10762444), and in further view of Shin et al. (hereinafter Shin, a non-patent literature reference titled “Continual Learning with Deep Generative Replay”).
Regarding Claim 1, Walters teaches a computer-implemented method (Walters, Par. [0011], “a method for detecting data drift”, thus a computer-implemented method is disclosed), the computer-implemented, method comprising:
identifying, by one or more processors, a generative synthetic data model corresponding to a historical training dataset […] wherein (i) the generative synthetic data model is stored […] (Walter, Par. [0028], “Dataset generator 103 can be configured to receive data models from model storage 109 or another component of system 100. Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113. As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset,” & Par. [0038], “When the performance of the trained synthetic data model satisfies performance criteria, model optimizer 107 can be configured to store the trained synthetic data model in model storage 109,” thus identifying a generative synthetic data model corresponding to a historical training dataset and storing the generative synthetic data model is disclosed, because Walters teaches a synthetic data model configured to generate data matching the statistical and content characteristics of a training dataset and further teaches storing the trained synthetic data model in model storage)
generating, by the one or more processors and using the generative synthetic data model, a synthetic dataset for the machine learning model (Walters, Par. [0041], “Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113 . As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109 . In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus generating, by one or more processors and using a generative synthetic data model, a synthetic dataset for a machine learning model is disclosed, because Walters teaches that dataset generator generates synthetic data and can generate such synthetic data using a data model without reliance on input data, where the data model generates data matching the statistical and content characteristics of a training dataset. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, dataset generator corresponds to the one or more processors, the data model corresponds to the generative synthetic data model, and the generated synthetic data / synthetic dataset corresponds to the synthetic dataset for the machine learning model)
[…] the synthetic dataset […] represented by the synthetic dataset (Walters, Par. [0028], “the data model can be configured to generate data matching statistical and content characteristics of a training dataset,” thus the historical training dataset being represented by the synthetic dataset is disclosed, because Walters teaches generating synthetic data that matches the statistical and content characteristics of the training dataset)
and initiating, by the one or more processors, the performance of one or more model performance-based operations based on the performance output for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift”, & Par. [0169], “In a random search, model optimizer 107 can be configured to select random coordinate points from the hyperparameter space and use the hyperparameters comprising these points to provision models. In some embodiments, model optimizer 107 can provision the computing resources with the new hyperparameters, without providing a new model. Instead, the computing resources can be configured to reset the model to the original state and retrain the model according to the new hyperparameters. Similarly, the computing resources can be configured to reuse or store the training data for the purpose of training multiple models”, thus initiating, by one or more processors, the performance of one or more model performance-based operations based on a performance output for a machine learning model is disclosed, because Walters teaches operations including training a model, hyperparameter tuning, and correcting the model based on detected data drift, such that the detected drift corresponds to the performance output and the training, tuning, and correction correspond to model performance-based operations. Walters further teaches that model optimizer can select hyperparameters, provision models using those hyperparameters, and cause computing resources to reset and retrain a model according to the new hyperparameters. Thus detected data drift corresponds to the performance output used to trigger action, and the training, hyperparameter tuning, correction, and retraining correspond to the one or more model performance-based operations)
Walters does not explicitly teach […] previously used to train a machine learning model, […] for use in evaluating post-deployment performance of the machine learning model […], generating, by the one or more processors, a performance output for the machine learning model based on a comparison between […] and a contemporary input dataset, wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset […], and […] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored.
However, Fly teaches:
[…] previously used to train a machine learning model, […] for use in evaluating post-deployment performance of the machine learning model […] (Fly, Par. [0013], “offline training environments are usually static and contain large amounts of historical data which is needed for the initial training of models. Once trained, the model algorithms (just a calculation, no longer containing any data) are then migrated into an online scoring environment,” & Par. [0014], “The connected systems identify or detect performance degradation (caused by drift) in an “online” operational scoring environment. Specifically, the inventive system identifies degradation by establishing a baseline in an “online” operational scoring environment of trained and validated modeling dataset and scores, and by ensuring that new online data and scores match those that were established in the “offline” discovery environment when the model was trained,” thus a historical training dataset previously used to train a machine learning model and use of the historical training dataset for evaluating post-deployment performance of the machine learning model is disclosed, because Fly teaches using historical data for initial training of the machine learning model, migrating the trained model to an online scoring environment, and subsequently evaluating performance degradation in the online scoring environment by comparing new online data and scores to a baseline established from the trained and validated modeling dataset and scores)
generating, by the one or more processors, a performance output for the machine learning model based on a comparison between […] and a contemporary input dataset, wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset […] (Fly, Par. [0018], “determine whether a drift exists between the scored requests and the training dataset based on distribution of scored requests relative to the batch scores in the training dataset, determine whether the drift is statistically significant, and generate a notification if drift is determined to be statistically significant,” & Par. [0021], “the configurable analysis time window represents the amount of time that is evaluated when comparing the real time events to the historical baseline to determine if drift is occurring,” thus generating a performance output based on a comparison involving contemporary input data, wherein the performance output indicates a data drift abnormality between the contemporary input data and the historical training data is disclosed, because Fly teaches comparing incoming real-time scoring requests to the historical training baseline, determining whether drift exists and whether the drift is statistically significant, and generating a notification when statistically significant drift is detected)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters’ synthetic-data comparison approach with Fly’s online drift-monitoring approach, because Fly teaches comparing live incoming online data against historical baseline information to detect drift, anomalies, and performance degradation, while Walters teaches using a synthetic dataset that preserves the statistical and content characteristics of historical/offline training data without directly exposing the original training data. Thus, a POSITA would have used Walters’ synthetic dataset in Fly’s framework as a privacy-preserving stand-in for the historical/offline training data, so that Fly’s incoming scoring requests could be compared against a synthetic historical representation rather than the original training dataset. In this way, incorporating Walters’ synthetic dataset into Fly’s framework would have enabled comparison of a synthetic historical representation with Fly’s live incoming scoring requests, such that performance output is generated from that comparison to evaluate drift. This would have reduced exposure of sensitive training data while facilitating use of training-data characteristics in the online environment (Walters, Par. [0048], "As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data)
Walters combined with Fly does not explicitly teach […] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored.
However, Shin teaches:
[…] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored (Shin, Page 8 – Section 4.3, “Because we assume that the past data are completely discarded, we trained the generator to mimic both current inputs and the generated samples from the previous generator. The generator thus reproduces cumulative input distribution of all encountered examples so far,” & Page 8 – Section 5, “The trained scholar model comprising a generator and a solver serves as a knowledge base of a task” & “the deep generative replay mechanism benefits from the fact that it maintains the former knowledge solely with input-target pairs produced from the saved networks,” thus using the generative synthetic data model in place of the historical training dataset and removing the historical training dataset after retaining the generative model is disclosed, because Shin teaches completely discarding the past data while retaining the trained generator/saved networks that reproduce and preserve the distribution and knowledge of the former training data)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Walters’s teaching of storing a generative synthetic data model and generating synthetic data representative of a training dataset, and Fly’s teaching of evaluating post-deployment model performance by comparing contemporary online data with a historical training-data baseline to detect drift, with Shin’s teaching of retaining prior-data information using a generative model while avoiding continued storage and replay of the actual past data. Therefore, a POSITA would have been motivated to incorporate Shin’s generative replay approach into the combined system of Walters and Fly so that the generative synthetic data model is retained in place of the historical training dataset and the historical training dataset is removed after the generative model is stored, thereby reducing the memory and storage requirements associated with retaining historical training data while preserving a representation of the historical data distribution for subsequent model performance evaluation and drift detection (Shin, Page 1 – Section 1, “a major drawback of memory-based approach is that it requires large working memory to store and replay past inputs. Moreover, such data storage and replay may not be viable in some real-world situations,” & Page 2 – Section 1, “We now propose an alternative approach to sequentially train deep neural networks without referring to past data. In our deep generative replay framework, the model retains previously acquired knowledge by the concurrent replay of generated pseudo-data. In particular, we train a deep generative model in the generative adversarial networks (GANs) framework [10] to mimic past data,” & “As deep generative replay supported by the scholar network retains the knowledge without revisiting actual past data, this framework can be employed to various practical situation involving privacy issues.”)
Regarding Claim 6, Walters and Fly combined with Shin teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the generative synthetic data model comprises a second machine learning model previously trained using the historical training dataset to generate the synthetic datasets representing the historical training dataset (Walters, Par. [0048], “This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data”, & Par. [0058], “In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus wherein the generative synthetic data model comprises a second machine learning model previously trained using the historical training dataset to generate the synthetic datasets representing the historical training dataset is disclosed, because Walters teaches that synthetic data can be generated using a synthetic dataset model that is generated using actual data, where the resulting synthetic data is similar to the actual data in values, value distributions, structure, and ordering, such that the synthetic dataset model corresponds to another machine learning model derived from the historical training data. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, Walters’ synthetic dataset model corresponds to another machine learning model previously trained using the historical training dataset, and the generated synthetic dataset corresponds to one or more synthetic datasets representing the historical training dataset)
Regarding Claim 7, Walters and Fly combined with Shin teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the generative synthetic data model is previously generated within a training time period corresponding to a training operation for the machine learning model (Walters, Par. [0129], “In some aspects, system 1300 (e.g., computing resources 1304 or model optimizer 1303 ) can be configured to store reference data stream data received from streaming data source 1301 for subsequent use as training data. In some embodiments, computing resources 1304 may have received the stored reference data stream data prior to beginning training of the new synthetic data model.”, and, “As an additional example, computing resources 1304 (or another component of system 1300 ) can be configured to gather data from streaming data source 1301 during a first time-interval (e.g., the prior repeat) and use this gathered data to train a new synthetic model in a subsequent time-interval (e.g., the current repeat). In various embodiments, computing resources 1304 can be configured to use the stored reference data stream data for training the new synthetic data model. In various embodiments, the training data can include both newly-received and stored data. When the synthetic data model is a Generative Adversarial Network, computing resources 1304 can be configured to train the new synthetic data model, in some embodiments, as described above with regard to FIGS. 8 and 9.”, thus wherein the generative synthetic data model is previously generated within a training time period corresponding to a training operation for the machine learning model is disclosed, because Walters teaches storing reference data stream data for subsequent use as training data and teaches that the stored reference data stream data may be received prior to beginning training of a new synthetic data model. Walters further teaches gathering data during a first time interval and using that gathered data to train a new synthetic model in a subsequent time interval, including use of stored reference data stream data for training the new synthetic data model. Thus, Walters teaches that the generative synthetic data model is generated during a training time period corresponding to training operations performed using historical and newly received training data)
Regarding Claim 9, Walters and Fly combined with Shin teaches all the limitations of claim 1 as cited above and Fly further teaches:
wherein the performance output is generated for the machine learning model at a data drift monitoring frequency (Fly, Par. [0022], “A configurable analysis time window may be used to compared the batch scored training dataset with the scored data, wherein the configurable analysis time window represents the amount of time that is evaluated when comparing the real time events to the historical baseline to determine if drift is occurring”, & Par. [113], “For example, the analysis window module 1125 may determine that a comparison can be made if the comparison window is equivalent to the training window. Per model, a configurable time window is established based on the variance tolerance and the velocity and volume of data. In other words, the analysis window represents the amount of time that will be evaluated when comparing the real time events to the historical baseline to determine if drift is occurring. The window is configurable by the user based on factors including: volume and velocity of data, anticipated rate of change of input data, and the risk associated with drift occurring and the immediacy of remediation.”, thus wherein the performance output is generated for the machine learning model at a data drift monitoring frequency is disclosed, because Fly teaches that a configurable analysis time window is used to compare scored data with a historical baseline to determine whether drift is occurring, and further teaches that per model, a configurable time window is established for that evaluation. Thus, Fly’s configurable analysis time window / per-model configurable time window corresponds to the data drift monitoring frequency, and Fly’s drift evaluation performed within that time window corresponds to generation of the respective performance output for the machine learning model)
Regarding Claim 10, Walters and Fly combined with Shin teaches all the limitations of claim 9 as cited above and Fly further teaches:
wherein the data drift monitoring frequency defines an evaluation time period and the contemporary input dataset comprises a plurality of input data objects corresponding to the evaluation time period (Fly, Par. [0022], “A configurable analysis time window may be used to compared the batch scored training dataset with the scored data, wherein the configurable analysis time window represents the amount of time that is evaluated when comparing the real time events to the historical baseline to determine if drift is occurring”, & Par. [0113], “For example, the analysis window module 1125 may determine that a comparison can be made if the comparison window is equivalent to the training window. Per model, a configurable time window is established based on the variance tolerance and the velocity and volume of data. In other words, the analysis window represents the amount of time that will be evaluated when comparing the real time events to the historical baseline to determine if drift is occurring. The window is configurable by the user based on factors including: volume and velocity of data, anticipated rate of change of input data, and the risk associated with drift occurring and the immediacy of remediation”, & Par. [0117], “The process also receives incoming scoring requests 1208 . Each scoring request is scored 1210 by collecting real time data inputs, and applying the machine learned models that are learned by the offline learning system. The newly scored online scores are thereafter logged as “test” results”, thus wherein the data drift monitoring frequency defines an evaluation time period and the contemporary input dataset comprises a plurality of input data objects corresponding to the evaluation time period is disclosed, because Fly teaches a configurable analysis time window that represents the amount of time that is evaluated when comparing real-time events to a historical baseline to determine whether drift is occurring, and further teaches that per model, a configurable time window is established for that evaluation. Fly also teaches receiving incoming scoring requests, where each scoring request is processed using real time data inputs and logged as test results. Thus, Fly’s configurable analysis time window / per-model configurable time window corresponds to the evaluation time period indicated by the data drift monitoring frequency, and Fly’s incoming scoring requests correspond to the plurality of input data objects corresponding to that evaluation time period)
Regarding Claim 11, Walters and Fly combined with Shin teaches all the limitations of claim 1 as cited above and Walters further teaches:
retraining the machine learning model in response to the performance output failing to satisfy a performance threshold for the machine learning model (Walters, Par. [0183], “data drift is detected based on a difference between a current data metric and a baseline data metric. For example, detecting data drift at step 1916 may involve determining whether a difference between a current data metric and a baseline data metric exceeds a threshold,” & Par. [0184], “At step 1918, the baseline model is updated based on the detected data drift… Updating the baseline model may include training the baseline model or performing hyperparameter tuning,” thus retraining the machine learning model in response to the performance output failing to satisfy a performance threshold is disclosed, because Walters teaches detecting drift when a current performance-related data metric differs from a baseline metric by more than a threshold and, in response to the detected drift, updating the model by training the model or performing hyperparameter tuning.)
Regarding Claim 12, Walters and Fly combined with Shin teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the model performance-based operation is initiated based on a comparison between the performance output and a performance threshold for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include receiving current data and detect data drift based on the data profile of the current data. In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift.”, & Par. [0209], “At step 2014 , data drift is detected based on at least one of a data metric, an updated model parameter, or an updated model hyperparameter. For example, data drift may be detected if a difference meets or exceeds a threshold difference in at least one of: a comparison of a generated data covariance matrix to a current data covariance matrix; a comparison of an updated model parameter to a previous model parameter; or a comparison of a model hyperparameter to a previous model hyperparameter. Detecting data drift at step 2014 may be based on a parameter threshold or a hyperparameter threshold as determined using process 1900 , for example. In some embodiments, model optimizer 107 may detect data drift in a manner consistent with the disclosed embodiments”, thus wherein the model performance-based operation is initiated based on a comparison between the performance output and a performance threshold for the machine learning model is disclosed, because Walters teaches detecting data drift based on differences in data, model parameters, or hyperparameters, and further teaches that such drift detection may occur when the difference meets or exceeds a threshold difference, including a parameter threshold or hyperparameter threshold. Walters also teaches operations including training, hyperparameter tuning, and correcting the model based on the detected data drift. Thus, Walters’ threshold difference / parameter threshold / hyperparameter threshold corresponds to the performance threshold, Walters’ drift-detection comparison corresponds to the comparison between the performance output and the performance threshold, and Walters’ training, tuning, and correcting correspond to the one or more model performance-based operations initiated based on that comparison)
Regarding Claim 13, Walters teaches:
identifying a generative synthetic data model corresponding to a historical training dataset […] wherein (i) the generative synthetic data model is stored […] (Walter, Par. [0028], “Dataset generator 103 can be configured to receive data models from model storage 109 or another component of system 100. Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113. As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset,” & Par. [0038], “When the performance of the trained synthetic data model satisfies performance criteria, model optimizer 107 can be configured to store the trained synthetic data model in model storage 109,” thus identifying a generative synthetic data model corresponding to a historical training dataset and storing the generative synthetic data model is disclosed, because Walters teaches a synthetic data model configured to generate data matching the statistical and content characteristics of a training dataset and further teaches storing the trained synthetic data model in model storage)
generating, using the generative synthetic data model, a synthetic dataset for the machine learning model (Walters, Par. [0041], “Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113 . As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109 . In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus generating, by one or more processors and using a generative synthetic data model, a synthetic dataset for a machine learning model is disclosed, because Walters teaches that dataset generator generates synthetic data and can generate such synthetic data using a data model without reliance on input data, where the data model generates data matching the statistical and content characteristics of a training dataset. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, dataset generator corresponds to the one or more processors, the data model corresponds to the generative synthetic data model, and the generated synthetic data / synthetic dataset corresponds to the synthetic dataset for the machine learning model)
[…] the synthetic dataset […] represented by the synthetic dataset (Walters, Par. [0028], “the data model can be configured to generate data matching statistical and content characteristics of a training dataset,” thus the historical training dataset being represented by the synthetic dataset is disclosed, because Walters teaches generating synthetic data that matches the statistical and content characteristics of the training dataset)
and initiating the performance of one or more model performance-based operations based on the performance output for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift”, & Par. [0169], “In a random search, model optimizer 107 can be configured to select random coordinate points from the hyperparameter space and use the hyperparameters comprising these points to provision models. In some embodiments, model optimizer 107 can provision the computing resources with the new hyperparameters, without providing a new model. Instead, the computing resources can be configured to reset the model to the original state and retrain the model according to the new hyperparameters. Similarly, the computing resources can be configured to reuse or store the training data for the purpose of training multiple models”, thus initiating, by one or more processors, the performance of one or more model performance-based operations based on a performance output for a machine learning model is disclosed, because Walters teaches operations including training a model, hyperparameter tuning, and correcting the model based on detected data drift, such that the detected drift corresponds to the performance output and the training, tuning, and correction correspond to model performance-based operations. Walters further teaches that model optimizer can select hyperparameters, provision models using those hyperparameters, and cause computing resources to reset and retrain a model according to the new hyperparameters. Thus detected data drift corresponds to the performance output used to trigger action, and the training, hyperparameter tuning, correction, and retraining correspond to the one or more model performance-based operations)
Walters does not explicitly teach […] previously used to train a machine learning model, […] for use in evaluating post-deployment performance of the machine learning model […], generating, by the one or more processors, a performance output for the machine learning model based on a comparison between […] and a contemporary input dataset, wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset […], and […] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored.
However, Fly teaches:
[…] previously used to train a machine learning model, […] for use in evaluating post-deployment performance of the machine learning model […] (Fly, Par. [0013], “offline training environments are usually static and contain large amounts of historical data which is needed for the initial training of models. Once trained, the model algorithms (just a calculation, no longer containing any data) are then migrated into an online scoring environment,” & Par. [0014], “The connected systems identify or detect performance degradation (caused by drift) in an “online” operational scoring environment. Specifically, the inventive system identifies degradation by establishing a baseline in an “online” operational scoring environment of trained and validated modeling dataset and scores, and by ensuring that new online data and scores match those that were established in the “offline” discovery environment when the model was trained,” thus a historical training dataset previously used to train a machine learning model and use of the historical training dataset for evaluating post-deployment performance of the machine learning model is disclosed, because Fly teaches using historical data for initial training of the machine learning model, migrating the trained model to an online scoring environment, and subsequently evaluating performance degradation in the online scoring environment by comparing new online data and scores to a baseline established from the trained and validated modeling dataset and scores)
generating a performance output for the machine learning model based on a comparison between […] and a contemporary input dataset, wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset […] (Fly, Par. [0018], “determine whether a drift exists between the scored requests and the training dataset based on distribution of scored requests relative to the batch scores in the training dataset, determine whether the drift is statistically significant, and generate a notification if drift is determined to be statistically significant,” & Par. [0021], “the configurable analysis time window represents the amount of time that is evaluated when comparing the real time events to the historical baseline to determine if drift is occurring,” thus generating a performance output based on a comparison involving contemporary input data, wherein the performance output indicates a data drift abnormality between the contemporary input data and the historical training data is disclosed, because Fly teaches comparing incoming real-time scoring requests to the historical training baseline, determining whether drift exists and whether the drift is statistically significant, and generating a notification when statistically significant drift is detected)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters’ synthetic-data comparison approach with Fly’s online drift-monitoring approach, because Fly teaches comparing live incoming online data against historical baseline information to detect drift, anomalies, and performance degradation, while Walters teaches using a synthetic dataset that preserves the statistical and content characteristics of historical/offline training data without directly exposing the original training data. Thus, a POSITA would have used Walters’ synthetic dataset in Fly’s framework as a privacy-preserving stand-in for the historical/offline training data, so that Fly’s incoming scoring requests could be compared against a synthetic historical representation rather than the original training dataset. In this way, incorporating Walters’ synthetic dataset into Fly’s framework would have enabled comparison of a synthetic historical representation with Fly’s live incoming scoring requests, such that performance output is generated from that comparison to evaluate drift. This would have reduced exposure of sensitive training data while facilitating use of training-data characteristics in the online environment (Walters, Par. [0048], "As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data)
Walters combined with Fly does not explicitly teach […] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored.
However, Shin teaches:
[…] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored (Shin, Page 8 – Section 4.3, “Because we assume that the past data are completely discarded, we trained the generator to mimic both current inputs and the generated samples from the previous generator. The generator thus reproduces cumulative input distribution of all encountered examples so far,” & Page 8 – Section 5, “The trained scholar model comprising a generator and a solver serves as a knowledge base of a task” & “the deep generative replay mechanism benefits from the fact that it maintains the former knowledge solely with input-target pairs produced from the saved networks,” thus using the generative synthetic data model in place of the historical training dataset and removing the historical training dataset after retaining the generative model is disclosed, because Shin teaches completely discarding the past data while retaining the trained generator/saved networks that reproduce and preserve the distribution and knowledge of the former training data)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Walters’s teaching of storing a generative synthetic data model and generating synthetic data representative of a training dataset, and Fly’s teaching of evaluating post-deployment model performance by comparing contemporary online data with a historical training-data baseline to detect drift, with Shin’s teaching of retaining prior-data information using a generative model while avoiding continued storage and replay of the actual past data. Therefore, a POSITA would have been motivated to incorporate Shin’s generative replay approach into the combined system of Walters and Fly so that the generative synthetic data model is retained in place of the historical training dataset and the historical training dataset is removed after the generative model is stored, thereby reducing the memory and storage requirements associated with retaining historical training data while preserving a representation of the historical data distribution for subsequent model performance evaluation and drift detection (Shin, Page 1 – Section 1, “a major drawback of memory-based approach is that it requires large working memory to store and replay past inputs. Moreover, such data storage and replay may not be viable in some real-world situations,” & Page 2 – Section 1, “We now propose an alternative approach to sequentially train deep neural networks without referring to past data. In our deep generative replay framework, the model retains previously acquired knowledge by the concurrent replay of generated pseudo-data. In particular, we train a deep generative model in the generative adversarial networks (GANs) framework [10] to mimic past data,” & “As deep generative replay supported by the scholar network retains the knowledge without revisiting actual past data, this framework can be employed to various practical situation involving privacy issues.”)
Regarding Claim 18, Walters teaches:
identifying a generative synthetic data model corresponding to a historical training dataset […] wherein (i) the generative synthetic data model is stored […] (Walter, Par. [0028], “Dataset generator 103 can be configured to receive data models from model storage 109 or another component of system 100. Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113. As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset,” & Par. [0038], “When the performance of the trained synthetic data model satisfies performance criteria, model optimizer 107 can be configured to store the trained synthetic data model in model storage 109,” thus identifying a generative synthetic data model corresponding to a historical training dataset and storing the generative synthetic data model is disclosed, because Walters teaches a synthetic data model configured to generate data matching the statistical and content characteristics of a training dataset and further teaches storing the trained synthetic data model in model storage)
generating, using the generative synthetic data model, a synthetic dataset for the machine learning model (Walters, Par. [0041], “Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113 . As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109 . In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus generating, by one or more processors and using a generative synthetic data model, a synthetic dataset for a machine learning model is disclosed, because Walters teaches that dataset generator generates synthetic data and can generate such synthetic data using a data model without reliance on input data, where the data model generates data matching the statistical and content characteristics of a training dataset. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, dataset generator corresponds to the one or more processors, the data model corresponds to the generative synthetic data model, and the generated synthetic data / synthetic dataset corresponds to the synthetic dataset for the machine learning model)
[…] the synthetic dataset […] represented by the synthetic dataset (Walters, Par. [0028], “the data model can be configured to generate data matching statistical and content characteristics of a training dataset,” thus the historical training dataset being represented by the synthetic dataset is disclosed, because Walters teaches generating synthetic data that matches the statistical and content characteristics of the training dataset)
and initiating the performance of one or more model performance-based operations based on the performance output for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift”, & Par. [0169], “In a random search, model optimizer 107 can be configured to select random coordinate points from the hyperparameter space and use the hyperparameters comprising these points to provision models. In some embodiments, model optimizer 107 can provision the computing resources with the new hyperparameters, without providing a new model. Instead, the computing resources can be configured to reset the model to the original state and retrain the model according to the new hyperparameters. Similarly, the computing resources can be configured to reuse or store the training data for the purpose of training multiple models”, thus initiating, by one or more processors, the performance of one or more model performance-based operations based on a performance output for a machine learning model is disclosed, because Walters teaches operations including training a model, hyperparameter tuning, and correcting the model based on detected data drift, such that the detected drift corresponds to the performance output and the training, tuning, and correction correspond to model performance-based operations. Walters further teaches that model optimizer can select hyperparameters, provision models using those hyperparameters, and cause computing resources to reset and retrain a model according to the new hyperparameters. Thus detected data drift corresponds to the performance output used to trigger action, and the training, hyperparameter tuning, correction, and retraining correspond to the one or more model performance-based operations)
Walters does not explicitly teach […] previously used to train a machine learning model, […] for use in evaluating post-deployment performance of the machine learning model […], generating, by the one or more processors, a performance output for the machine learning model based on a comparison between […] and a contemporary input dataset, wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset […], and […] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored.
However, Fly teaches:
[…] previously used to train a machine learning model, […] for use in evaluating post-deployment performance of the machine learning model […] (Fly, Par. [0013], “offline training environments are usually static and contain large amounts of historical data which is needed for the initial training of models. Once trained, the model algorithms (just a calculation, no longer containing any data) are then migrated into an online scoring environment,” & Par. [0014], “The connected systems identify or detect performance degradation (caused by drift) in an “online” operational scoring environment. Specifically, the inventive system identifies degradation by establishing a baseline in an “online” operational scoring environment of trained and validated modeling dataset and scores, and by ensuring that new online data and scores match those that were established in the “offline” discovery environment when the model was trained,” thus a historical training dataset previously used to train a machine learning model and use of the historical training dataset for evaluating post-deployment performance of the machine learning model is disclosed, because Fly teaches using historical data for initial training of the machine learning model, migrating the trained model to an online scoring environment, and subsequently evaluating performance degradation in the online scoring environment by comparing new online data and scores to a baseline established from the trained and validated modeling dataset and scores)
generating a performance output for the machine learning model based on a comparison between […] and a contemporary input dataset, wherein the performance output indicates a data drift abnormality between the contemporary input dataset and the historical training dataset […] (Fly, Par. [0018], “determine whether a drift exists between the scored requests and the training dataset based on distribution of scored requests relative to the batch scores in the training dataset, determine whether the drift is statistically significant, and generate a notification if drift is determined to be statistically significant,” & Par. [0021], “the configurable analysis time window represents the amount of time that is evaluated when comparing the real time events to the historical baseline to determine if drift is occurring,” thus generating a performance output based on a comparison involving contemporary input data, wherein the performance output indicates a data drift abnormality between the contemporary input data and the historical training data is disclosed, because Fly teaches comparing incoming real-time scoring requests to the historical training baseline, determining whether drift exists and whether the drift is statistically significant, and generating a notification when statistically significant drift is detected)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters’ synthetic-data comparison approach with Fly’s online drift-monitoring approach, because Fly teaches comparing live incoming online data against historical baseline information to detect drift, anomalies, and performance degradation, while Walters teaches using a synthetic dataset that preserves the statistical and content characteristics of historical/offline training data without directly exposing the original training data. Thus, a POSITA would have used Walters’ synthetic dataset in Fly’s framework as a privacy-preserving stand-in for the historical/offline training data, so that Fly’s incoming scoring requests could be compared against a synthetic historical representation rather than the original training dataset. In this way, incorporating Walters’ synthetic dataset into Fly’s framework would have enabled comparison of a synthetic historical representation with Fly’s live incoming scoring requests, such that performance output is generated from that comparison to evaluate drift. This would have reduced exposure of sensitive training data while facilitating use of training-data characteristics in the online environment (Walters, Par. [0048], "As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data)
Walters combined with Fly does not explicitly teach […] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored.
However, Shin teaches:
[…] in place of […the historical training dataset…] […]and (ii) […the historical training dataset…] is removed from storage after […the generative synthetic data model…] is stored (Shin, Page 8 – Section 4.3, “Because we assume that the past data are completely discarded, we trained the generator to mimic both current inputs and the generated samples from the previous generator. The generator thus reproduces cumulative input distribution of all encountered examples so far,” & Page 8 – Section 5, “The trained scholar model comprising a generator and a solver serves as a knowledge base of a task” & “the deep generative replay mechanism benefits from the fact that it maintains the former knowledge solely with input-target pairs produced from the saved networks,” thus using the generative synthetic data model in place of the historical training dataset and removing the historical training dataset after retaining the generative model is disclosed, because Shin teaches completely discarding the past data while retaining the trained generator/saved networks that reproduce and preserve the distribution and knowledge of the former training data)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Walters’s teaching of storing a generative synthetic data model and generating synthetic data representative of a training dataset, and Fly’s teaching of evaluating post-deployment model performance by comparing contemporary online data with a historical training-data baseline to detect drift, with Shin’s teaching of retaining prior-data information using a generative model while avoiding continued storage and replay of the actual past data. Therefore, a POSITA would have been motivated to incorporate Shin’s generative replay approach into the combined system of Walters and Fly so that the generative synthetic data model is retained in place of the historical training dataset and the historical training dataset is removed after the generative model is stored, thereby reducing the memory and storage requirements associated with retaining historical training data while preserving a representation of the historical data distribution for subsequent model performance evaluation and drift detection (Shin, Page 1 – Section 1, “a major drawback of memory-based approach is that it requires large working memory to store and replay past inputs. Moreover, such data storage and replay may not be viable in some real-world situations,” & Page 2 – Section 1, “We now propose an alternative approach to sequentially train deep neural networks without referring to past data. In our deep generative replay framework, the model retains previously acquired knowledge by the concurrent replay of generated pseudo-data. In particular, we train a deep generative model in the generative adversarial networks (GANs) framework [10] to mimic past data,” & “As deep generative replay supported by the scholar network retains the knowledge without revisiting actual past data, this framework can be employed to various practical situation involving privacy issues.”)
Regarding Claim 19, Walters and Fly combined with Shin teaches all the limitations of claim 18 as cited above and Walters further teaches:
wherein the generative synthetic data model comprises a second machine learning model previously trained using the historical training dataset to generate the synthetic dataset representing the historical training dataset (Walters, Par. [0048], “This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data”, & Par. [0058], “In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus wherein the generative synthetic data model comprises a second machine learning model previously trained using the historical training dataset to generate the synthetic dataset representing the historical training dataset is disclosed, because Walters teaches that synthetic data can be generated using a synthetic dataset model that is generated using actual data, where the resulting synthetic data is similar to the actual data in values, value distributions, structure, and ordering, such that the synthetic dataset model corresponds to another machine learning model derived from the historical training data. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, Walters’ synthetic dataset model corresponds to another machine learning model previously trained using the historical training dataset, and the generated synthetic dataset corresponds to one or more synthetic datasets representing the historical training dataset)
Regarding Claim 20, Walters and Fly combined with Shin teaches all the limitations of claim 18 as cited above and Walters further teaches:
wherein the generative synthetic data model is previously generated within a training time period corresponding to a training operation for the machine learning model (Walters, Par. [0129], “In some aspects, system 1300 (e.g., computing resources 1304 or model optimizer 1303 ) can be configured to store reference data stream data received from streaming data source 1301 for subsequent use as training data. In some embodiments, computing resources 1304 may have received the stored reference data stream data prior to beginning training of the new synthetic data model.”, and, “As an additional example, computing resources 1304 (or another component of system 1300 ) can be configured to gather data from streaming data source 1301 during a first time-interval (e.g., the prior repeat) and use this gathered data to train a new synthetic model in a subsequent time-interval (e.g., the current repeat). In various embodiments, computing resources 1304 can be configured to use the stored reference data stream data for training the new synthetic data model. In various embodiments, the training data can include both newly-received and stored data. When the synthetic data model is a Generative Adversarial Network, computing resources 1304 can be configured to train the new synthetic data model, in some embodiments, as described above with regard to FIGS. 8 and 9.”, thus wherein the generative synthetic data model is previously generated within a training time period corresponding to a training operation for the machine learning model is disclosed, because Walters teaches storing reference data stream data for subsequent use as training data and teaches that the stored reference data stream data may be received prior to beginning training of a new synthetic data model. Walters further teaches gathering data during a first time interval and using that gathered data to train a new synthetic model in a subsequent time interval, including use of stored reference data stream data for training the new synthetic data model. Thus, Walters teaches that the generative synthetic data model is generated during a training time period corresponding to training operations performed using historical and newly received training data)
Regarding Claim 21, Walters and Fly combined with Shin teaches all the limitations of claim 13 as cited above and Walters further teaches:
retraining the machine learning model in response to the performance output failing to satisfy a performance threshold for the machine learning model (Walters, Par. [0183], “data drift is detected based on a difference between a current data metric and a baseline data metric. For example, detecting data drift at step 1916 may involve determining whether a difference between a current data metric and a baseline data metric exceeds a threshold,” & Par. [0184], “At step 1918, the baseline model is updated based on the detected data drift… Updating the baseline model may include training the baseline model or performing hyperparameter tuning,” thus retraining the machine learning model in response to the performance output failing to satisfy a performance threshold is disclosed, because Walters teaches detecting drift when a current performance-related data metric differs from a baseline metric by more than a threshold and, in response to the detected drift, updating the model by training the model or performing hyperparameter tuning.)
Claims 2-5 and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Walters et al. (hereafter Walters) (US 10599957), in view of Fly et al. (hereinafter Fly) (US 10762444), in view of Shin et al. (hereinafter Shin, a non-patent literature reference titled “Continual Learning with Deep Generative Replay”), and in further in view of Chen et al. (hereinafter Chen), a non-patent literature reference titled “Developments in MLflow: A system to accelerate the machine learning lifecycle”).
Regarding Claim 2, Walters and Fly combined with Shin teaches all of the limitations of claim 1 as cited above and Walters further teaches:
wherein […]corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system. This information can be provided using interface 113 . The databases can include cloud-based databases (e.g., AMAZON WEB SERVICES S3 buckets) or on-premises databases. The information can include model information, such as the type and/or purpose of the model and any measures of classification error”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107”, thus wherein […]corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model is disclosed, because Walters teaches that model storage stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage and can be provided from that storage in response to a request from dataset generator or model optimizer. Thus, model storage corresponds to the model registry, the stored synthetic data model and descriptive information correspond to information indicative of the generative synthetic data model, and the stored data models and model information correspond to storage in association with the machine learning model)
Walters and Fly combined with Shin does not explicitly teach […] a data model representation […].
However, Chen teaches […] a data model representation […] (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus a data model representation is disclosed, because Chen teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a representation of model-related information in a stored format. Thus, Chen’s bundled serialized model format and associated dependency information correspond to the data model representation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters, Fly, and Shin with Chen, because Walters and Fly teach storing model related information, including information indicative of a generative synthetic data model, within registry like model storage associated with a machine learning model, while Chen teaches a Model Registry framework for managing the model deployment lifecycle and standardizing model handling through cataloguing, lineage tracking, and versioned model management. Incorporating Chen’s model representation approach into the Walters and Fly’s system would have allowed the stored generative synthetic model related information to be maintained in a more structured and reusable form within the model registry, thereby improving organization, lifecycle management, and reducing the likelihood of broken or inferior models being used in deployment (Chen, Page 1 – Section 1, “While traditional software has a well-defined set of product features to be built, ML development revolves around experimentation: ML developers constantly experiment with new datasets, models, software libraries, tuning parameters, etc. to optimize a metric such as model accuracy. Because model performance depends heavily on the input data and training process, reproducibility is paramount throughout ML development”, & Page 3 – Section 4.1, “To address these needs, we introduced the MLflow Model Registry: a collaborative hub for managing the model deployment lifecycle. In addition to providing cataloguing and lineage tracking capabilities, the Model Registry standardizes the model deployment workflow by enabling ML developers and deployment engineers to version their models and transition them through four logical stages: "Development," "Staging," "Production," and "Archived." Further, organizations can restrict access to particular stages on a per-user or per-role basis, and the Model Registry enables users to request stage transitions from colleagues. These stages structure the deployment process and provide a model review framework, guarding against common pitfalls, such as the deployment of broken or inferior models to production”)
Regarding Claim 3, Walters, Fly, and Shin combined with Chen teaches all the limitations of claim 2 as cited above and Walters further teaches:
wherein the model registry comprises […] comprises a respective machine learning model and […] corresponding to a respective generative synthetic data model corresponding to the respective machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103”, thus Walters teaches wherein the model registry comprises a respective machine learning model and information corresponding to a respective generative synthetic data model corresponding to the respective machine learning model, because Walters teaches that model storage 109 stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage 109 and provided from that storage in response to a request. Thus, model storage 109 corresponds to the model registry, the stored data models correspond to the respective machine learning model, and the stored synthetic data model together with its associated stored information corresponds to information indicative of a respective generative synthetic data model corresponding to the respective machine learning model)
Chen further teaches […] a plurality of composite model data objects and a composite model data object of the plurality of composite model data objects comprises […] a respective data model representation […] (Chen, Page 2 – Section 3.1, “MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles”, & Page 2 – Section 4.1, “In particular, several adopters in the IoT domain described use cases wherein a unique model is built for each independent device or entity. These training processes frequently produce tens of thousands or even hundreds of thousands of models. ML practitioners and deployment engineers must vet each of these models, map them to specific production applications, and deploy them systematically to guard against regressions and breakages”, & Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus Chen teaches a plurality of composite model data objects and each of the plurality of composite model data objects comprises a respective data model representation, because Chen teaches that the MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles, which corresponds to maintaining multiple model related objects in a registry, and further teaches use cases involving tens of thousands or hundreds of thousands of models, which corresponds to a plurality of such model related objects. Chen also teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a respective data model representation associated with each model object. Thus, Chen’s Model Registry corresponds to storage of a plurality of composite model data objects, the multiple catalogued models correspond to the plurality of such objects, and the bundled serialized model format and additional dependency information correspond to the respective data model representation)
Regarding Claim 4, Walters, Fly, and Shin combined with Chen teaches all the limitations of claim 2 as cited above and Chen further teaches:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus wherein the data model representation comprises a serialized representation of the generative synthetic data model is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to representing the generative synthetic data model in serialized form. Thus, Chen’s serialized model format corresponds to the serialized representation of the generative synthetic data model)
Regarding Claim 5, Walters, Fly, and Shin combined with Chen teaches all the limitations of claim 4 as cited above and Chen further teaches:
generating the generative synthetic data model by deserializing the data model representation (Chen, Page 2 – Section 2, “ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, & Page 3 – Section 4.1, “The Model Registry integrates with these continuous integration and deployment (CI/CD) tools by providing APIs for fetching models by version and stage that developers can incorporate into their existing CI/CD workflows”, thus generating the generative synthetic data model by deserializing the data model representation is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to storing the model in serialized form, and further teaches fetching models by version and stage for use in CI/CD workflows, which corresponds to retrieving and loading the serialized model for use. Thus, Chen’s serialized model format corresponds to the data model representation, and Chen’s fetching and loading of the serialized model corresponds to generating the generative synthetic data model by deserializing the data model representation)
Walters further teaches generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset (Walters, Par. [0041], “As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “In step 305 , in some embodiments, dataset generator 103 can generate synthetic data. Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109”, thus generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset is disclosed, because Walters teaches that dataset generator generates synthetic data using a data model without reliance on input data, where the data model is configured to generate data matching the statistical and content characteristics of a training dataset and may map from a random or pseudorandom vector to elements in the training data space. Walters further teaches that dataset generator generates synthetic data. Thus, Walters’ generated synthetic data corresponds to the plurality of evaluation samples, and the generated data matching the statistical and content characteristics of the training dataset corresponds to evaluation samples corresponding to the historical training dataset)
Regarding Claim 14, Walters, Fly, and Shin combined with Chen teaches all of the limitations of claim 13 as cited above and Walters further teaches:
wherein […]corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system. This information can be provided using interface 113 . The databases can include cloud-based databases (e.g., AMAZON WEB SERVICES S3 buckets) or on-premises databases. The information can include model information, such as the type and/or purpose of the model and any measures of classification error”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107”, thus wherein […]corresponding to the generative synthetic data model is stored within a model registry in association with the machine learning model is disclosed, because Walters teaches that model storage stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage and can be provided from that storage in response to a request from dataset generator or model optimizer. Thus, model storage corresponds to the model registry, the stored synthetic data model and descriptive information correspond to information indicative of the generative synthetic data model, and the stored data models and model information correspond to storage in association with the machine learning model)
Walters and Fly combined with Shin does not explicitly teach […] a data model representation […].
However, Chen teaches […] a data model representation […] (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus a data model representation is disclosed, because Chen teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a representation of model-related information in a stored format. Thus, Chen’s bundled serialized model format and associated dependency information correspond to the data model representation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters and Fly with Chen, because Walters and Fly teach storing model related information, including information indicative of a generative synthetic data model, within registry like model storage associated with a machine learning model, while Chen teaches a Model Registry framework for managing the model deployment lifecycle and standardizing model handling through cataloguing, lineage tracking, and versioned model management. Incorporating Chen’s model representation approach into the Walters and Fly’s system would have allowed the stored generative synthetic model related information to be maintained in a more structured and reusable form within the model registry, thereby improving organization, lifecycle management, and reducing the likelihood of broken or inferior models being used in deployment (Chen, Page 1 – Section 1, “While traditional software has a well-defined set of product features to be built, ML development revolves around experimentation: ML developers constantly experiment with new datasets, models, software libraries, tuning parameters, etc. to optimize a metric such as model accuracy. Because model performance depends heavily on the input data and training process, reproducibility is paramount throughout ML development”, & Page 3 – Section 4.1, “To address these needs, we introduced the MLflow Model Registry: a collaborative hub for managing the model deployment lifecycle. In addition to providing cataloguing and lineage tracking capabilities, the Model Registry standardizes the model deployment workflow by enabling ML developers and deployment engineers to version their models and transition them through four logical stages: "Development," "Staging," "Production," and "Archived." Further, organizations can restrict access to particular stages on a per-user or per-role basis, and the Model Registry enables users to request stage transitions from colleagues. These stages structure the deployment process and provide a model review framework, guarding against common pitfalls, such as the deployment of broken or inferior models to production”)
Regarding Claim 15, Walters, Fly, and Shin combined with Chen teaches all the limitations of claim 14 as cited above and Walters further teaches:
wherein the model registry comprises […] comprises a respective machine learning model and […] corresponding to a respective generative synthetic data model corresponding to the respective machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103”, thus wherein the model registry comprises […] comprises a respective machine learning model and […] corresponding to a respective generative synthetic data model corresponding to the respective machine learning model is disclosed, because Walters teaches that model storage 109 stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage 109 and provided from that storage in response to a request. Thus, model storage 109 corresponds to the model registry, the stored data models correspond to the respective machine learning model, and the stored synthetic data model together with its associated stored information corresponds to information indicative of a respective generative synthetic data model corresponding to the respective machine learning model)
Chen further teaches […] a plurality of composite model data objects and a composite model data object of the plurality of composite model data objects comprises […] a respective data model representation […] (Chen, Page 2 – Section 3.1, “MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles”, & Page 2 – Section 4.1, “In particular, several adopters in the IoT domain described use cases wherein a unique model is built for each independent device or entity. These training processes frequently produce tens of thousands or even hundreds of thousands of models. ML practitioners and deployment engineers must vet each of these models, map them to specific production applications, and deploy them systematically to guard against regressions and breakages”, & Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus Chen teaches a plurality of composite model data objects and each of the plurality of composite model data objects comprises a respective data model representation, because Chen teaches that the MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles, which corresponds to maintaining multiple model related objects in a registry, and further teaches use cases involving tens of thousands or hundreds of thousands of models, which corresponds to a plurality of such model related objects. Chen also teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a respective data model representation associated with each model object. Thus, Chen’s Model Registry corresponds to storage of a plurality of composite model data objects, the multiple catalogued models correspond to the plurality of such objects, and the bundled serialized model format and additional dependency information correspond to the respective data model representation)
Regarding Claim 16, Walters, Fly, and Shin combined with Chen teaches all the limitations of claim 14 as cited above and Chen further teaches:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus wherein the data model representation comprises a serialized representation of the generative synthetic data model is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to representing the generative synthetic data model in serialized form. Thus, Chen’s serialized model format corresponds to the serialized representation of the generative synthetic data model)
Regarding Claim 17, Walters, Fly, and Shin combined with Chen teaches all the limitations of claim 16 as cited above and Chen further teaches:
generating the generative synthetic data model by deserializing the data model representation (Chen, Page 2 – Section 2, “ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, & Page 3 – Section 4.1, “The Model Registry integrates with these continuous integration and deployment (CI/CD) tools by providing APIs for fetching models by version and stage that developers can incorporate into their existing CI/CD workflows”, thus generating the generative synthetic data model by deserializing the data model representation is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to storing the model in serialized form, and further teaches fetching models by version and stage for use in CI/CD workflows, which corresponds to retrieving and loading the serialized model for use. Thus, Chen’s serialized model format corresponds to the data model representation, and Chen’s fetching and loading of the serialized model corresponds to generating the generative synthetic data model by deserializing the data model representation)
Walters further teaches generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset (Walters, Par. [0041], “As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “In step 305 , in some embodiments, dataset generator 103 can generate synthetic data. Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109”, thus generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset is disclosed, because Walters teaches that dataset generator generates synthetic data using a data model without reliance on input data, where the data model is configured to generate data matching the statistical and content characteristics of a training dataset and may map from a random or pseudorandom vector to elements in the training data space. Walters further teaches that dataset generator generates synthetic data. Thus, Walters’ generated synthetic data corresponds to the plurality of evaluation samples, and the generated data matching the statistical and content characteristics of the training dataset corresponds to evaluation samples corresponding to the historical training dataset)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHLIET ADMASU whose telephone number is (571)272-0034. The examiner can normally be reached Mon-Fri, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.T.A./
Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights