DETAILED ACTION
This communication is in response to the Application No. 18/365,364 filed August 04, 2023
in which Claims 1 - 20 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: the following paragraphs are duplicates:
Paragraph [0013] and [0034].
Paragraph [0014] and [0035].
Paragraph [0051] and [0078].
Paragraph [0054] and [0091].
appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 are rejected under 35 U.S.C. 101 because these claimed inventions are directed to an
abstract idea without significantly more.
Regarding Claim 1:
Step 1: Claim 1 is a method type claim. Therefore, Claims 1-12 fall within one of the four statutory
categories (i.e., process, machine, manufacture, or composition of matter).
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance
of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable
interpretation, covers performance of the limitation by mathematical calculation but for the recitation
of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract
ideas.
Identifying, […], a generative synthetic data model corresponding to a historical training dataset for a machine learning model (mental process – identifying a generative synthetic data model may be performed mentally a user observing/analyzing the historical training dataset and accordingly using judgement/evaluation to identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model based on said analysis)
generating […] a synthetic dataset for the machine learning model (mental process – generating a synthetic dataset may be performed mentally by a user observing/analyzing the machine learning model and accordingly using judgement/evaluation to generate a synthetic dataset for the machine learning model based on said analysis)
Generating […] a performance output for the machine learning model based on a comparison between the synthetic dataset and a contemporary input dataset (mental process – generating a performance output for the machine learning model may be performed mentally by a user observing/analyzing the synthetic dataset and the contemporary input dataset and accordingly using judgement/evaluation to generate a performance output for the machine learning model based on said analysis)
and initiating […] the performance of one or more model performance-based operations based on the performance output for the machine learning model (mental process – initiating the performance of one or more model performance-based operations may be performed mentally by a user observing/analyzing the performance output for the machine learning model and using judgement/evaluation to decide whether to initiate a responsive operation based on said analysis. For example, as stated in the specification (Par. [0067]), a model performance-based operation may include initiating the performance of one or more alerts, messages, instructions, and/or the like in response to a performance output.)
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
Step 2B: The claim does not include additional elements considered individually and in combination that
are sufficient to amount to significantly more than the judicial exception.
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…]by one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 1 - 12. The additional limitations of the dependent claims are addressed below.
Regarding Claim 2:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 2 depends on.
Step 2A Prong 2 & Step 2B:
wherein a data model representation indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that a data model representation indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 3:
Step 2A Prong 1: See the rejection of Claim 2 above, which Claim 3 depends on.
Step 2A Prong 2 & Step 2B:
wherein the model registry comprises a plurality of composite model data objects and each of the plurality of composite model data objects comprises a respective machine learning model and a respective data model representation indicative of a respective generative synthetic data model corresponding to the respective machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the model registry comprising a plurality of composite model data objects and each of the plurality of composite model data objects comprising a respective machine learning model and a respective data model representation indicative of a respective generative synthetic data model corresponding to the respective machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 2. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 4:
Step 2A Prong 1: See the rejection of Claim 2 above, which Claim 4 depends on.
Step 2A Prong 2 & Step 2B:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the data model representation comprising a serialized representation of the generative synthetic data model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 2. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 2A Prong 1: See the rejection of Claim 4 above, which Claim 5 depends on.
generating the generative synthetic data model by deserializing the data model representation(mental process – generating the generative synthetic data model may be performed mentally by a user observing/analyzing the data model representation and using judgment/evaluation to reconstruct a model from that representation based on said analysis. For example, the specification explains that a generative synthetic data model may include a statistical model, such as a linear regression model or tree-based model, and a user can formulate a simple statistical model using pen and paper (Par. [0055]))
And generating […] a plurality of evaluation samples corresponding to the historical training dataset (mental process – generating a plurality of evaluation samples may be performed mentally by a user observing/analyzing the historical training data and accordingly using judgment/evaluation to generate a plurality of evaluation samples corresponding to the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
[…], using the generative synthetic data model, […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 4. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 6:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 6 depends on.
[…]to generate one or more synthetic datasets representing the historical training dataset (mental process – generate one or more synthetic datasets may be performed mentally by a user observing/analyzing the historical training dataset and using judgment/evaluation to generate one or more synthetic datasets representing the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
wherein the generative synthetic data model comprises another machine learning model previously trained using the historical training dataset […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a trained machine learning model without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 7:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 7 depends on.
Step 2A Prong 2 & Step 2B:
wherein the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 8:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 8 depends on.
Step 2A Prong 2 & Step 2B:
wherein the performance output is indicative of a predicted data drift between the historical training dataset and the contemporary input dataset (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the performance output is indicative of a predicted data drift between the historical training dataset and the contemporary input dataset does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 9:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 9 depends on.
Step 2A Prong 2 & Step 2B:
wherein a respective performance output is generated for the machine learning model at a data drift monitoring frequency (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that a respective performance output is generated for the machine learning model at a data drift monitoring frequency does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 10:
Step 2A Prong 1: See the rejection of Claim 9 above, which Claim 10 depends on.
Step 2A Prong 2 & Step 2B:
wherein the data drift monitoring frequency is indicative of an evaluation time period and the contemporary input dataset comprises a plurality of input data objects corresponding to the evaluation time period (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the data drift monitoring frequency is indicative of an evaluation time period and the contemporary input dataset comprising a plurality of input data objects corresponding to the evaluation time period does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 9. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 11:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 11 depends on.
Step 2A Prong 2 & Step 2B:
wherein the one or more model performance-based operations comprise one or more model retraining operations using the contemporary input dataset (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of model retraining using the contemporary input dataset without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 12:
Step 2A Prong 1: See the rejection of Claim 1 above, which Claim 12 depends on.
Step 2A Prong 2 & Step 2B:
wherein the one or more model performance-based operations are initiated based on a comparison between the performance output and a performance threshold for the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the one or more model performance-based operations are initiated based on a comparison between the performance output and a performance threshold does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 1. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 13:
Step 1: Claim 13 is a system type claim. Therefore, Claims 13-17 fall within one of the four statutory
categories (i.e., process, machine, manufacture, or composition of matter).
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance
of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable
interpretation, covers performance of the limitation by mathematical calculation but for the recitation
of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract
ideas.
Identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model (mental process – identifying a generative synthetic data model may be performed mentally a user observing/analyzing the historical training dataset and accordingly using judgement/evaluation to identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model based on said analysis)
generate […] a synthetic dataset for the machine learning model (mental process – generating a synthetic dataset may be performed mentally by a user observing/analyzing the machine learning model and accordingly using judgement/evaluation to generate a synthetic dataset for the machine learning model based on said analysis)
generate a performance output for the machine learning model based on a comparison between the synthetic dataset and a contemporary input dataset (mental process – generating a performance output for the machine learning model may be performed mentally by a user observing/analyzing the synthetic dataset and the contemporary input dataset and accordingly using judgement/evaluation to generate a performance output for the machine learning model based on said analysis)
and initiate the performance of one or more model performance-based operations based on the performance output for the machine learning model (mental process – initiating the performance of one or more model performance-based operations may be performed mentally by a user observing/analyzing the performance output for the machine learning model and using judgement/evaluation to decide whether to initiate a responsive operation based on said analysis. For example, as stated in the specification (Par. [0067]), a model performance-based operation may include initiating the performance of one or more alerts, messages, instructions, and/or the like in response to a performance output.)
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
[…]memory and one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
Step 2B: The claim does not include additional elements considered individually and in combination that
are sufficient to amount to significantly more than the judicial exception.
[…]memory and one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
For the reasons above, Claim 13 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 13 - 17. The additional limitations of the dependent claims are addressed below.
Regarding Claim 14:
Step 2A Prong 1: See the rejection of Claim 13 above, which Claim 14 depends on.
Step 2A Prong 2 & Step 2B:
wherein a data model representation indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that a data model representation indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 13. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 15:
Step 2A Prong 1: See the rejection of Claim 14 above, which Claim 15 depends on.
Step 2A Prong 2 & Step 2B:
wherein the model registry comprises a plurality of composite model data objects and each of the plurality of composite model data objects comprises a respective machine learning model and a respective data model representation indicative of a respective generative synthetic data model corresponding to the respective machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the model registry comprising a plurality of composite model data objects and each of the plurality of composite model data objects comprising a respective machine learning model and a respective data model representation indicative of a respective generative synthetic data model corresponding to the respective machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 14. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 16:
Step 2A Prong 1: See the rejection of Claim 14 above, which Claim 16 depends on.
Step 2A Prong 2 & Step 2B:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the data model representation comprising a serialized representation of the generative synthetic data model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 14. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 17:
Step 2A Prong 1: See the rejection of Claim 16 above, which Claim 17 depends on.
generating the generative synthetic data model by deserializing the data model representation (mental process – generating the generative synthetic data model may be performed mentally by a user observing/analyzing the data model representation and using judgment/evaluation to reconstruct a model from that representation based on said analysis. For example, the specification explains that a generative synthetic data model may include a statistical model, such as a linear regression model or tree-based model, and a user can formulate a simple statistical model using pen and paper (Par. [0055]))
And generating […] a plurality of evaluation samples corresponding to the historical training dataset (mental process – generating a plurality of evaluation samples may be performed mentally by a user observing/analyzing the historical training data and accordingly using judgment/evaluation to generate a plurality of evaluation samples corresponding to the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
[…], using the generative synthetic data model, […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 16. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 18:
Step 1: Claim 18 is a non-transitory computer-readable storage media type claim. Therefore, Claims 18-20 fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance
of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable
interpretation, covers performance of the limitation by mathematical calculation but for the recitation
of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract
ideas.
Identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model (mental process – identifying a generative synthetic data model may be performed mentally a user observing/analyzing the historical training dataset and accordingly using judgement/evaluation to identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model based on said analysis)
generate […] a synthetic dataset for the machine learning model (mental process – generating a synthetic dataset may be performed mentally by a user observing/analyzing the machine learning model and accordingly using judgement/evaluation to generate a synthetic dataset for the machine learning model based on said analysis)
generate a performance output for the machine learning model based on a comparison between the synthetic dataset and a contemporary input dataset (mental process – generating a performance output for the machine learning model may be performed mentally by a user observing/analyzing the synthetic dataset and the contemporary input dataset and accordingly using judgement/evaluation to generate a performance output for the machine learning model based on said analysis)
and initiate the performance of one or more model performance-based operations based on the performance output for the machine learning model (mental process – initiating the performance of one or more model performance-based operations may be performed mentally by a user observing/analyzing the performance output for the machine learning model and using judgement/evaluation to decide whether to initiate a responsive operation based on said analysis. For example, as stated in the specification (Par. [0067]), a model performance-based operation may include initiating the performance of one or more alerts, messages, instructions, and/or the like in response to a performance output.)
Step 2A Prong 2: This judicial exception is not integrated into a practical application.
[…]one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
Step 2B: The claim does not include additional elements considered individually and in combination that
are sufficient to amount to significantly more than the judicial exception.
[…]memory and one or more processors[…] (recited at a high-level of generality (i.e., a generic processor, and memory) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
[…] and using the generative synthetic data model[…] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a machine learning model/ the generative synthetic data model(Par. [0055], “In addition, or alternatively, a generative synthetic data model may include a machine learning model.”) without significantly more)
For the reasons above, Claim 18 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 18 - 20. The additional limitations of the dependent claims are addressed below.
Regarding Claim 19:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 19 depends on.
[…]to generate one or more synthetic datasets representing the historical training dataset (mental process – generate one or more synthetic datasets may be performed mentally by a user observing/analyzing the historical training dataset and using judgment/evaluation to generate one or more synthetic datasets representing the historical training dataset based on said analysis)
Step 2A Prong 2 & Step 2B: This judicial exception is not integrated into a practical application.
wherein the generative synthetic data model comprises another machine learning model previously trained using the historical training dataset […] (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: high level recitation of using a trained machine learning model without significantly more)
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 18. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Regarding Claim 20:
Step 2A Prong 1: See the rejection of Claim 18 above, which Claim 20 depends on.
Step 2A Prong 2 & Step 2B:
wherein the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, this additional element does not integrate the
abstract idea into practical application because it does not impose any meaningful limits on practicing
the abstract idea, as discussed above in the rejection of claim 18. The claim does not include additional
elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-12, 13, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Walters et al. (hereafter Walters) (US 10599957), in view of Fly et al. (hereinafter Fly) (US 10762444).
Regarding Claim 1, Walters teaches a computer-implemented method (Walters, Par. [0011], “a method for detecting data drift”, thus a computer-implemented method is disclosed), the computer-implemented, method comprising:
identifying, by one or more processors, a generative synthetic data model corresponding to a historical training dataset for a machine learning model (Walters, Par. [0048], “The data model can be generated using synthetic data in some aspects. This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data. As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data.”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107 , or another component of system 100”, thus identifying, by one or more processors, a generative synthetic data model corresponding to a historical training dataset for a machine learning model is disclosed, because Walters teaches that a synthetic dataset model can be generated from actual data, where the resulting synthetic data is similar to the actual data in values, value distributions, structure, and ordering, such that the synthetic dataset model corresponds to the historical training data used for the machine learning application. Walters further teaches that the system can receive a synthetic data model from model storage in response to a request, which corresponds to identifying the generative synthetic data model for use by the processor. Thus, Walters’ synthetic dataset model corresponds to the generative synthetic data model, Walters’ actual data corresponds to the historical training dataset, and Walters’ machine learning application corresponds to the machine learning model)
generating, by the one or more processors and using the generative synthetic data model, a synthetic dataset for the machine learning model (Walters, Par. [0041], “Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113 . As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109 . In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus generating, by one or more processors and using a generative synthetic data model, a synthetic dataset for a machine learning model is disclosed, because Walters teaches that dataset generator generates synthetic data and can generate such synthetic data using a data model without reliance on input data, where the data model generates data matching the statistical and content characteristics of a training dataset. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, dataset generator corresponds to the one or more processors, the data model corresponds to the generative synthetic data model, and the generated synthetic data / synthetic dataset corresponds to the synthetic dataset for the machine learning model)
generating, by the one or more processors, a performance output for the machine learning model based on a comparison between the synthetic dataset and […] (Walters, Par. [0098], “Process 900 can then proceed to step 907 . In step 907 , system 100 (e.g., model optimizer 107 , computational resources 101 , or the like) can determine a similarity metric value using the normalized reference dataset and the synthetic training dataset. System 100 can be configured to generate the similarity metric value according to a similarity metric. In some aspects, the similarity metric value can include at least one of a statistical correlation score (e.g., a score dependent on the covariances or univariate distributions of the synthetic data and the normalized reference dataset), a data similarity score (e.g., a score dependent on a number of matching or similar elements in the synthetic dataset and normalized reference dataset), or data quality score (e.g., a score dependent on at least one of a number of duplicate elements in each of the synthetic dataset and normalized reference dataset, a prevalence of the most common value in each of the synthetic dataset and normalized reference dataset, a maximum difference of rare values in each of the synthetic dataset and normalized reference dataset, the differences in schema between the synthetic dataset and normalized reference dataset, or the like). System 100 can be configured to calculate these scores using the synthetic dataset and a reference dataset”, & Par. [0168], “In various embodiments, the development instance can be configured to evaluate the performance of the trained model. The development instance can evaluate the performance of the trained model according to a performance metric, as described herein. In some embodiments, the value of the performance metric can depend on a similarity between data generated by a trained model and the training data used to train the trained model. In various embodiments, the value of the performance metric can depend on an accuracy of classifications or predictions output by the trained model”, thus generating, by one or more processors, a performance output for a machine learning model based on a comparison between a synthetic dataset and another dataset is disclosed, because Walters teaches that system determines and generates a similarity metric value using the normalized reference dataset and the synthetic training dataset, where the similarity metric value may include a statistical correlation score, a data similarity score, or a data quality score calculated using the synthetic dataset and the reference dataset. Walters further teaches that a development instance evaluates the performance of the trained model according to a performance metric, where the value of the performance metric can depend on a similarity between data generated by the trained model and the training data used to train the trained model. Thus, the similarity metric value / performance metric corresponds to the performance output, the synthetic training dataset corresponds to the synthetic dataset, and the normalized reference dataset / training data used to train the model corresponds to the dataset compared against the synthetic dataset)
and initiating, by the one or more processors, the performance of one or more model performance-based operations based on the performance output for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift”, & Par. [0169], “In a random search, model optimizer 107 can be configured to select random coordinate points from the hyperparameter space and use the hyperparameters comprising these points to provision models. In some embodiments, model optimizer 107 can provision the computing resources with the new hyperparameters, without providing a new model. Instead, the computing resources can be configured to reset the model to the original state and retrain the model according to the new hyperparameters. Similarly, the computing resources can be configured to reuse or store the training data for the purpose of training multiple models”, thus initiating, by one or more processors, the performance of one or more model performance-based operations based on a performance output for a machine learning model is disclosed, because Walters teaches operations including training a model, hyperparameter tuning, and correcting the model based on detected data drift, such that the detected drift corresponds to the performance output and the training, tuning, and correction correspond to model performance-based operations. Walters further teaches that model optimizer can select hyperparameters, provision models using those hyperparameters, and cause computing resources to reset and retrain a model according to the new hyperparameters. Thus detected data drift corresponds to the performance output used to trigger action, and the training, hyperparameter tuning, correction, and retraining correspond to the one or more model performance-based operations)
Walters does not explicitly teach […] a contemporary input dataset.
However, Fly teaches […] a contemporary input dataset (Fly, Par. [0020], “In one embodiment, the process for determining whether drift exists is comprised of identifying the percentage of records in the incoming scoring requests and the offline training dataset that are based on a variable, calculating a difference in the two percentages, calculating the natural log of percentages of records that are based on the variable in the incoming scoring requests and offline training dataset, and combine the calculated differences in the two percentages and the calculated natural logs”, & Par. [0023], “when executed by a processor, cause the processor to monitor an online scoring environment, wherein the online scoring environment is comprised of machine learning models for scoring incoming requests, the incoming requests being received and thereafter scored by using model learning models, and wherein outcome actions are initiated based on the score received by each incoming request, perform statistical analysis on incoming scoring requests to determine whether incoming scoring requests are anomalous, determine whether the scores output in the online scoring environment are anomalous”, thus Fly teaches a contemporary input dataset because Fly describes incoming scoring requests and incoming requests that are received and scored in an online scoring environment, and further compares those incoming scoring requests against the offline training dataset to determine whether drift exists. Thus, Fly’s incoming scoring requests / incoming requests correspond to contemporary input data used for current model evaluation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters’ synthetic-data comparison approach with Fly’s online drift-monitoring approach, because Fly teaches comparing live incoming online data against historical baseline information to detect drift, anomalies, and performance degradation, while Walters teaches using a synthetic dataset that preserves the statistical and content characteristics of historical/offline training data without directly exposing the original training data. Thus, a POSITA would have used Walters’ synthetic dataset in Fly’s framework as a privacy-preserving stand-in for the historical/offline training data, so that Fly’s incoming scoring requests could be compared against a synthetic historical representation rather than the original training dataset. In this way, incorporating Walters’ synthetic dataset into Fly’s framework would have enabled comparison of a synthetic historical representation with Fly’s live incoming scoring requests, such that performance output is generated from that comparison to evaluate drift. This would have reduced exposure of sensitive training data while facilitating use of training-data characteristics in the online environment (Walters, Par. [0048], "As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data)
Regarding Claim 6, Walters combined with Fly teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the generative synthetic data model comprises another machine learning model previously trained using the historical training dataset to generate one or more synthetic datasets representing the historical training dataset (Walters, Par. [0048], “This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data”, & Par. [0058], “In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus wherein the generative synthetic data model comprises another machine learning model previously trained using the historical training dataset to generate one or more synthetic datasets representing the historical training dataset is disclosed, because Walters teaches that synthetic data can be generated using a synthetic dataset model that is generated using actual data, where the resulting synthetic data is similar to the actual data in values, value distributions, structure, and ordering, such that the synthetic dataset model corresponds to another machine learning model derived from the historical training data. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, Walters’ synthetic dataset model corresponds to another machine learning model previously trained using the historical training dataset, and the generated synthetic dataset corresponds to one or more synthetic datasets representing the historical training dataset)
Regarding Claim 7, Walters combined with Fly teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model (Walters, Par. [0129], “In some aspects, system 1300 (e.g., computing resources 1304 or model optimizer 1303 ) can be configured to store reference data stream data received from streaming data source 1301 for subsequent use as training data. In some embodiments, computing resources 1304 may have received the stored reference data stream data prior to beginning training of the new synthetic data model.”, and, “As an additional example, computing resources 1304 (or another component of system 1300 ) can be configured to gather data from streaming data source 1301 during a first time-interval (e.g., the prior repeat) and use this gathered data to train a new synthetic model in a subsequent time-interval (e.g., the current repeat). In various embodiments, computing resources 1304 can be configured to use the stored reference data stream data for training the new synthetic data model. In various embodiments, the training data can include both newly-received and stored data. When the synthetic data model is a Generative Adversarial Network, computing resources 1304 can be configured to train the new synthetic data model, in some embodiments, as described above with regard to FIGS. 8 and 9.”, thus wherein the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model is disclosed, because Walters teaches storing reference data stream data for subsequent use as training data and teaches that the stored reference data stream data may be received prior to beginning training of a new synthetic data model. Walters further teaches gathering data during a first time interval and using that gathered data to train a new synthetic model in a subsequent time interval, including use of stored reference data stream data for training the new synthetic data model. Thus, Walters teaches that the generative synthetic data model is generated during a training time period corresponding to training operations performed using historical and newly received training data)
Regarding Claim 8, Walters combined with Fly teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the performance output is indicative of a predicted data drift between the historical training dataset and the contemporary input dataset (Walters, Par. [0098], “System 100 can be configured to generate the similarity metric value according to a similarity metric. In some aspects, the similarity metric value can include at least one of a statistical correlation score (e.g., a score dependent on the covariances or univariate distributions of the synthetic data and the normalized reference dataset), a data similarity score (e.g., a score dependent on a number of matching or similar elements in the synthetic dataset and normalized reference dataset), or data quality score”, Par. [0010], “The operations may include receiving model input data and generating predicted data using the predictive model, based on the model input data. The operations may include receiving event data and detecting data drift based on the predicted data and the event data. In some embodiments, the operations include receiving current data and detect data drift based on the data profile of the current data. In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter.”, thus wherein the performance output is indicative of a predicted data drift between the historical training dataset and the contemporary input dataset is disclosed, because Walters teaches generating a similarity metric value such as a statistical correlation score, data similarity score, or data quality score based on comparison of the synthetic data and the normalized reference dataset, which reflects difference from historical data, and further teaches receiving current data and detecting data drift based on current data, predicted data, or event data. Thus, Walters’ detected data drift corresponds to predicted data drift, and Walters’ similarity metric value corresponds to the performance output indicative of that predicted data drift)
Regarding Claim 9, Walters combined with Fly teaches all the limitations of claim 1 as cited above and Fly further teaches:
wherein a respective performance output is generated for the machine learning model at a data drift monitoring frequency (Fly, Par. [0022], “A configurable analysis time window may be used to compared the batch scored training dataset with the scored data, wherein the configurable analysis time window represents the amount of time that is evaluated when comparing the real time events to the historical baseline to determine if drift is occurring”, & Par. [113], “For example, the analysis window module 1125 may determine that a comparison can be made if the comparison window is equivalent to the training window. Per model, a configurable time window is established based on the variance tolerance and the velocity and volume of data. In other words, the analysis window represents the amount of time that will be evaluated when comparing the real time events to the historical baseline to determine if drift is occurring. The window is configurable by the user based on factors including: volume and velocity of data, anticipated rate of change of input data, and the risk associated with drift occurring and the immediacy of remediation.”, thus wherein a respective performance output is generated for the machine learning model at a data drift monitoring frequency is disclosed, because Fly teaches that a configurable analysis time window is used to compare scored data with a historical baseline to determine whether drift is occurring, and further teaches that per model, a configurable time window is established for that evaluation. Thus, Fly’s configurable analysis time window / per-model configurable time window corresponds to the data drift monitoring frequency, and Fly’s drift evaluation performed within that time window corresponds to generation of the respective performance output for the machine learning model)
Regarding Claim 10, Walters combined with Fly teaches all the limitations of claim 9 as cited above and Fly further teaches:
wherein the data drift monitoring frequency is indicative of an evaluation time period and the contemporary input dataset comprises a plurality of input data objects corresponding to the evaluation time period (Fly, Par. [0022], “A configurable analysis time window may be used to compared the batch scored training dataset with the scored data, wherein the configurable analysis time window represents the amount of time that is evaluated when comparing the real time events to the historical baseline to determine if drift is occurring”, & Par. [0113], “For example, the analysis window module 1125 may determine that a comparison can be made if the comparison window is equivalent to the training window. Per model, a configurable time window is established based on the variance tolerance and the velocity and volume of data. In other words, the analysis window represents the amount of time that will be evaluated when comparing the real time events to the historical baseline to determine if drift is occurring. The window is configurable by the user based on factors including: volume and velocity of data, anticipated rate of change of input data, and the risk associated with drift occurring and the immediacy of remediation”, & Par. [0117], “The process also receives incoming scoring requests 1208 . Each scoring request is scored 1210 by collecting real time data inputs, and applying the machine learned models that are learned by the offline learning system. The newly scored online scores are thereafter logged as “test” results”, thus wherein the data drift monitoring frequency is indicative of an evaluation time period and the contemporary input dataset comprises a plurality of input data objects corresponding to the evaluation time period is disclosed, because Fly teaches a configurable analysis time window that represents the amount of time that is evaluated when comparing real-time events to a historical baseline to determine whether drift is occurring, and further teaches that per model, a configurable time window is established for that evaluation. Fly also teaches receiving incoming scoring requests, where each scoring request is processed using real time data inputs and logged as test results. Thus, Fly’s configurable analysis time window / per-model configurable time window corresponds to the evaluation time period indicated by the data drift monitoring frequency, and Fly’s incoming scoring requests correspond to the plurality of input data objects corresponding to that evaluation time period)
Regarding Claim 11, Walters combined with Fly teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the one or more model performance-based operations comprise one or more model retraining operations using the contemporary input dataset (Walters, Par. [0007], “Further, in situations where a synthetic data model or predictive model is retrained using new data, the model may continue to provide accurate results Nevertheless, the model may drift over time as it is retrained on data that has drifted. The model may experience shifts in model parameters or hyperparameters, making data drift may be difficult for users to detect. Further, when models are provided to users, the users may be unable to train or update the model based on drift without further intervention from the model provider, so the user depends on the provider to provide updated models. Therefore, systems and methods are needed to detect data drift, correct synthetic data models or predictive models for drift, and notify downstream users”, & Par. [0010], “In some embodiments, the operations include receiving current data and detect data drift based on the data profile of the current data. In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift”, thus wherein the one or more model performance-based operations comprise one or more model retraining operations using the contemporary input dataset is disclosed, because Walters teaches that synthetic data models or predictive models may be retrained using new data, and further teaches receiving current data, detecting data drift based on the data profile of the current data, and correcting the model based on the detected data drift. Thus, Walters’ retraining of the model using new data corresponds to the one or more model retraining operations using the contemporary input dataset.)
Regarding Claim 12, Walters combined with Fly teaches all the limitations of claim 1 as cited above and Walters further teaches:
wherein the one or more model performance-based operations are initiated based on a comparison between the performance output and a performance threshold for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include receiving current data and detect data drift based on the data profile of the current data. In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift.”, & Par. [0209], “At step 2014 , data drift is detected based on at least one of a data metric, an updated model parameter, or an updated model hyperparameter. For example, data drift may be detected if a difference meets or exceeds a threshold difference in at least one of: a comparison of a generated data covariance matrix to a current data covariance matrix; a comparison of an updated model parameter to a previous model parameter; or a comparison of a model hyperparameter to a previous model hyperparameter. Detecting data drift at step 2014 may be based on a parameter threshold or a hyperparameter threshold as determined using process 1900 , for example. In some embodiments, model optimizer 107 may detect data drift in a manner consistent with the disclosed embodiments”, thus wherein the one or more model performance-based operations are initiated based on a comparison between the performance output and a performance threshold for the machine learning model is disclosed, because Walters teaches detecting data drift based on differences in data, model parameters, or hyperparameters, and further teaches that such drift detection may occur when the difference meets or exceeds a threshold difference, including a parameter threshold or hyperparameter threshold. Walters also teaches operations including training, hyperparameter tuning, and correcting the model based on the detected data drift. Thus, Walters’ threshold difference / parameter threshold / hyperparameter threshold corresponds to the performance threshold, Walters’ drift-detection comparison corresponds to the comparison between the performance output and the performance threshold, and Walters’ training, tuning, and correcting correspond to the one or more model performance-based operations initiated based on that comparison)
Regarding Claim 13, Walters teaches a computing system (Walters, Par. [0010], “a system configured to perform operations for detecting data drift”, thus a computing system is disclosed) comprising memory (Walters, Par. [0010], “the system may include one or more memory units”, thus memory is disclosed) and one or more processors communicatively coupled to the memory, the one or more processors configured to:
identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model (Walters, Par. [0048], “The data model can be generated using synthetic data in some aspects. This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data. As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data.”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107 , or another component of system 100”, thus identifying a generative synthetic data model corresponding to a historical training dataset for a machine learning model is disclosed, because Walters teaches that a synthetic dataset model can be generated from actual data, where the resulting synthetic data is similar to the actual data in values, value distributions, structure, and ordering, such that the synthetic dataset model corresponds to the historical training data used for the machine learning application. Walters further teaches that the system can receive a synthetic data model from model storage in response to a request, which corresponds to identifying the generative synthetic data model. Thus, Walters’ synthetic dataset model corresponds to the generative synthetic data model, Walters’ actual data corresponds to the historical training dataset, and Walters’ machine learning application corresponds to the machine learning model)
generate, using the generative synthetic data model, a synthetic dataset for the machine learning model (Walters, Par. [0041], “Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113 . As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109 . In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus generating, using a generative synthetic data model, a synthetic dataset for a machine learning model is disclosed, because Walters teaches that dataset generator generates synthetic data and can generate such synthetic data using a data model without reliance on input data, where the data model generates data matching the statistical and content characteristics of a training dataset. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, the data model corresponds to the generative synthetic data model, and the generated synthetic data / synthetic dataset corresponds to the synthetic dataset for the machine learning model)
generate a performance output for the machine learning model based on a comparison between the synthetic dataset and […] (Walters, Par. [0098], “Process 900 can then proceed to step 907 . In step 907 , system 100 (e.g., model optimizer 107 , computational resources 101 , or the like) can determine a similarity metric value using the normalized reference dataset and the synthetic training dataset. System 100 can be configured to generate the similarity metric value according to a similarity metric. In some aspects, the similarity metric value can include at least one of a statistical correlation score (e.g., a score dependent on the covariances or univariate distributions of the synthetic data and the normalized reference dataset), a data similarity score (e.g., a score dependent on a number of matching or similar elements in the synthetic dataset and normalized reference dataset), or data quality score (e.g., a score dependent on at least one of a number of duplicate elements in each of the synthetic dataset and normalized reference dataset, a prevalence of the most common value in each of the synthetic dataset and normalized reference dataset, a maximum difference of rare values in each of the synthetic dataset and normalized reference dataset, the differences in schema between the synthetic dataset and normalized reference dataset, or the like). System 100 can be configured to calculate these scores using the synthetic dataset and a reference dataset”, & Par. [0168], “In various embodiments, the development instance can be configured to evaluate the performance of the trained model. The development instance can evaluate the performance of the trained model according to a performance metric, as described herein. In some embodiments, the value of the performance metric can depend on a similarity between data generated by a trained model and the training data used to train the trained model. In various embodiments, the value of the performance metric can depend on an accuracy of classifications or predictions output by the trained model”, thus generate a performance output for a machine learning model based on a comparison between a synthetic dataset and another dataset is disclosed, because Walters teaches that system determines and generates a similarity metric value using the normalized reference dataset and the synthetic training dataset, where the similarity metric value may include a statistical correlation score, a data similarity score, or a data quality score calculated using the synthetic dataset and the reference dataset. Walters further teaches that a development instance evaluates the performance of the trained model according to a performance metric, where the value of the performance metric can depend on a similarity between data generated by the trained model and the training data used to train the trained model. Thus, the similarity metric value / performance metric corresponds to the performance output, the synthetic training dataset corresponds to the synthetic dataset, and the normalized reference dataset / training data used to train the model corresponds to the dataset compared against the synthetic dataset)
and initiate the performance of one or more model performance-based operations based on the performance output for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift”, & Par. [0169], “In a random search, model optimizer 107 can be configured to select random coordinate points from the hyperparameter space and use the hyperparameters comprising these points to provision models. In some embodiments, model optimizer 107 can provision the computing resources with the new hyperparameters, without providing a new model. Instead, the computing resources can be configured to reset the model to the original state and retrain the model according to the new hyperparameters. Similarly, the computing resources can be configured to reuse or store the training data for the purpose of training multiple models”, thus initiating the performance of one or more model performance-based operations based on a performance output for a machine learning model is disclosed, because Walters teaches operations including training a model, hyperparameter tuning, and correcting the model based on detected data drift, such that the detected drift corresponds to the performance output and the training, tuning, and correction correspond to model performance-based operations. Walters further teaches that model optimizer can select hyperparameters, provision models using those hyperparameters, and cause computing resources to reset and retrain a model according to the new hyperparameters. Thus detected data drift corresponds to the performance output used to trigger action, and the training, hyperparameter tuning, correction, and retraining correspond to the one or more model performance-based operations)
Walters does not explicitly teach […] a contemporary input dataset.
However, Fly teaches […] a contemporary input dataset (Fly, Par. [0020], “In one embodiment, the process for determining whether drift exists is comprised of identifying the percentage of records in the incoming scoring requests and the offline training dataset that are based on a variable, calculating a difference in the two percentages, calculating the natural log of percentages of records that are based on the variable in the incoming scoring requests and offline training dataset, and combine the calculated differences in the two percentages and the calculated natural logs”, & Par. [0023], “when executed by a processor, cause the processor to monitor an online scoring environment, wherein the online scoring environment is comprised of machine learning models for scoring incoming requests, the incoming requests being received and thereafter scored by using model learning models, and wherein outcome actions are initiated based on the score received by each incoming request, perform statistical analysis on incoming scoring requests to determine whether incoming scoring requests are anomalous, determine whether the scores output in the online scoring environment are anomalous”, thus Fly teaches a contemporary input dataset because Fly describes incoming scoring requests and incoming requests that are received and scored in an online scoring environment, and further compares those incoming scoring requests against the offline training dataset to determine whether drift exists. Thus, Fly’s incoming scoring requests / incoming requests correspond to contemporary input data used for current model evaluation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters’ synthetic-data comparison approach with Fly’s online drift-monitoring approach, because Fly teaches comparing live incoming online data against historical baseline information to detect drift, anomalies, and performance degradation, while Walters teaches using a synthetic dataset that preserves the statistical and content characteristics of historical/offline training data without directly exposing the original training data. Thus, a POSITA would have used Walters’ synthetic dataset in Fly’s framework as a privacy-preserving stand-in for the historical/offline training data, so that Fly’s incoming scoring requests could be compared against a synthetic historical representation rather than the original training dataset. In this way, incorporating Walters’ synthetic dataset into Fly’s framework would have enabled comparison of a synthetic historical representation with Fly’s live incoming scoring requests, such that performance output is generated from that comparison to evaluate drift. This would have reduced exposure of sensitive training data while facilitating use of training-data characteristics in the online environment (Walters, Par. [0048], "As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data)
Regarding Claim 18, Walters teaches one or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors (Walters, Par. [0012], “non-transitory computer readable storage media may store program instructions, which are executed by at least one processor device”, thus one or more non-transitory computer-readable storage media is disclosed), cause the one or more processors to:
identify a generative synthetic data model corresponding to a historical training dataset for a machine learning model (Walters, Par. [0048], “The data model can be generated using synthetic data in some aspects. This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data. As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data.”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107 , or another component of system 100”, thus identifying a generative synthetic data model corresponding to a historical training dataset for a machine learning model is disclosed, because Walters teaches that a synthetic dataset model can be generated from actual data, where the resulting synthetic data is similar to the actual data in values, value distributions, structure, and ordering, such that the synthetic dataset model corresponds to the historical training data used for the machine learning application. Walters further teaches that the system can receive a synthetic data model from model storage in response to a request, which corresponds to identifying the generative synthetic data model. Thus, Walters’ synthetic dataset model corresponds to the generative synthetic data model, Walters’ actual data corresponds to the historical training dataset, and Walters’ machine learning application corresponds to the machine learning model)
generate, using the generative synthetic data model, a synthetic dataset for the machine learning model (Walters, Par. [0041], “Dataset generator 103 can be configured to generate synthetic data. For example, dataset generator 103 can be configured to generate synthetic data by identifying and replacing sensitive information in data received from database 103 or interface 113 . As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109 . In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus generating, using a generative synthetic data model, a synthetic dataset for a machine learning model is disclosed, because Walters teaches that dataset generator generates synthetic data and can generate such synthetic data using a data model without reliance on input data, where the data model generates data matching the statistical and content characteristics of a training dataset. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, the data model corresponds to the generative synthetic data model, and the generated synthetic data / synthetic dataset corresponds to the synthetic dataset for the machine learning model)
generate a performance output for the machine learning model based on a comparison between the synthetic dataset and […] (Walters, Par. [0098], “Process 900 can then proceed to step 907 . In step 907 , system 100 (e.g., model optimizer 107 , computational resources 101 , or the like) can determine a similarity metric value using the normalized reference dataset and the synthetic training dataset. System 100 can be configured to generate the similarity metric value according to a similarity metric. In some aspects, the similarity metric value can include at least one of a statistical correlation score (e.g., a score dependent on the covariances or univariate distributions of the synthetic data and the normalized reference dataset), a data similarity score (e.g., a score dependent on a number of matching or similar elements in the synthetic dataset and normalized reference dataset), or data quality score (e.g., a score dependent on at least one of a number of duplicate elements in each of the synthetic dataset and normalized reference dataset, a prevalence of the most common value in each of the synthetic dataset and normalized reference dataset, a maximum difference of rare values in each of the synthetic dataset and normalized reference dataset, the differences in schema between the synthetic dataset and normalized reference dataset, or the like). System 100 can be configured to calculate these scores using the synthetic dataset and a reference dataset”, & Par. [0168], “In various embodiments, the development instance can be configured to evaluate the performance of the trained model. The development instance can evaluate the performance of the trained model according to a performance metric, as described herein. In some embodiments, the value of the performance metric can depend on a similarity between data generated by a trained model and the training data used to train the trained model. In various embodiments, the value of the performance metric can depend on an accuracy of classifications or predictions output by the trained model”, thus generate a performance output for a machine learning model based on a comparison between a synthetic dataset and another dataset is disclosed, because Walters teaches that system determines and generates a similarity metric value using the normalized reference dataset and the synthetic training dataset, where the similarity metric value may include a statistical correlation score, a data similarity score, or a data quality score calculated using the synthetic dataset and the reference dataset. Walters further teaches that a development instance evaluates the performance of the trained model according to a performance metric, where the value of the performance metric can depend on a similarity between data generated by the trained model and the training data used to train the trained model. Thus, the similarity metric value / performance metric corresponds to the performance output, the synthetic training dataset corresponds to the synthetic dataset, and the normalized reference dataset / training data used to train the model corresponds to the dataset compared against the synthetic dataset)
and initiate the performance of one or more model performance-based operations based on the performance output for the machine learning model (Walters, Par. [0010], “In some embodiments, the operations include training a model training and detecting data drift based on a difference in a trained model parameter from a baseline model parameter. The operations may include hyperparameter tuning and detecting data drift based on a difference in a tuned hyperparameter from a baseline hyperparameter. The operations may include correcting the model based on the detected data drift”, & Par. [0169], “In a random search, model optimizer 107 can be configured to select random coordinate points from the hyperparameter space and use the hyperparameters comprising these points to provision models. In some embodiments, model optimizer 107 can provision the computing resources with the new hyperparameters, without providing a new model. Instead, the computing resources can be configured to reset the model to the original state and retrain the model according to the new hyperparameters. Similarly, the computing resources can be configured to reuse or store the training data for the purpose of training multiple models”, thus initiating the performance of one or more model performance-based operations based on a performance output for a machine learning model is disclosed, because Walters teaches operations including training a model, hyperparameter tuning, and correcting the model based on detected data drift, such that the detected drift corresponds to the performance output and the training, tuning, and correction correspond to model performance-based operations. Walters further teaches that model optimizer can select hyperparameters, provision models using those hyperparameters, and cause computing resources to reset and retrain a model according to the new hyperparameters. Thus detected data drift corresponds to the performance output used to trigger action, and the training, hyperparameter tuning, correction, and retraining correspond to the one or more model performance-based operations)
Walters does not explicitly teach […] a contemporary input dataset.
However, Fly teaches […] a contemporary input dataset (Fly, Par. [0020], “In one embodiment, the process for determining whether drift exists is comprised of identifying the percentage of records in the incoming scoring requests and the offline training dataset that are based on a variable, calculating a difference in the two percentages, calculating the natural log of percentages of records that are based on the variable in the incoming scoring requests and offline training dataset, and combine the calculated differences in the two percentages and the calculated natural logs”, & Par. [0023], “when executed by a processor, cause the processor to monitor an online scoring environment, wherein the online scoring environment is comprised of machine learning models for scoring incoming requests, the incoming requests being received and thereafter scored by using model learning models, and wherein outcome actions are initiated based on the score received by each incoming request, perform statistical analysis on incoming scoring requests to determine whether incoming scoring requests are anomalous, determine whether the scores output in the online scoring environment are anomalous”, thus Fly teaches a contemporary input dataset because Fly describes incoming scoring requests and incoming requests that are received and scored in an online scoring environment, and further compares those incoming scoring requests against the offline training dataset to determine whether drift exists. Thus, Fly’s incoming scoring requests / incoming requests correspond to contemporary input data used for current model evaluation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters’ synthetic-data comparison approach with Fly’s online drift-monitoring approach, because Fly teaches comparing live incoming online data against historical baseline information to detect drift, anomalies, and performance degradation, while Walters teaches using a synthetic dataset that preserves the statistical and content characteristics of historical/offline training data without directly exposing the original training data. Thus, a POSITA would have used Walters’ synthetic dataset in Fly’s framework as a privacy-preserving stand-in for the historical/offline training data, so that Fly’s incoming scoring requests could be compared against a synthetic historical representation rather than the original training dataset. In this way, incorporating Walters’ synthetic dataset into Fly’s framework would have enabled comparison of a synthetic historical representation with Fly’s live incoming scoring requests, such that performance output is generated from that comparison to evaluate drift. This would have reduced exposure of sensitive training data while facilitating use of training-data characteristics in the online environment (Walters, Par. [0048], "As the actual data may include sensitive information, and generating the data model may require distribution and/or review of training data, the use of the synthetic data can protect the privacy and security of the entities and/or individuals whose activities are recorded by the actual data)
Regarding Claim 19, Walters combined with Fly teaches all the limitations of claim 18 as cited above and Walters further teaches:
wherein the generative synthetic data model comprises another machine learning model previously trained using the historical training dataset to generate one or more synthetic datasets representing the historical training dataset (Walters, Par. [0048], “This synthetic data can be generated using a synthetic dataset model, which can in turn be generated using actual data. The synthetic data may be similar to the actual data in terms of values, value distributions (e.g., univariate and multivariate statistics of the synthetic data may be similar to that of the actual data), structure and ordering, or the like. In this manner, the data model for the machine learning application can be generated without directly using the actual data”, & Par. [0058], “In some embodiments, dataset generator 103 can be configured to identify sensitive data items using a recurrent neural network. Dataset generator 103 can be configured to use the data model retrieved from model storage 109 to generate a synthetic dataset by replacing the sensitive data items with synthetic data items”, thus wherein the generative synthetic data model comprises another machine learning model previously trained using the historical training dataset to generate one or more synthetic datasets representing the historical training dataset is disclosed, because Walters teaches that synthetic data can be generated using a synthetic dataset model that is generated using actual data, where the resulting synthetic data is similar to the actual data in values, value distributions, structure, and ordering, such that the synthetic dataset model corresponds to another machine learning model derived from the historical training data. Walters further teaches that dataset generator uses the data model retrieved from model storage to generate a synthetic dataset. Thus, Walters’ synthetic dataset model corresponds to another machine learning model previously trained using the historical training dataset, and the generated synthetic dataset corresponds to one or more synthetic datasets representing the historical training dataset)
Regarding Claim 20, Walters combined with Fly teaches all the limitations of claim 18 as cited above and Walters further teaches:
wherein the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model (Walters, Par. [0129], “In some aspects, system 1300 (e.g., computing resources 1304 or model optimizer 1303 ) can be configured to store reference data stream data received from streaming data source 1301 for subsequent use as training data. In some embodiments, computing resources 1304 may have received the stored reference data stream data prior to beginning training of the new synthetic data model.”, and, “As an additional example, computing resources 1304 (or another component of system 1300 ) can be configured to gather data from streaming data source 1301 during a first time-interval (e.g., the prior repeat) and use this gathered data to train a new synthetic model in a subsequent time-interval (e.g., the current repeat). In various embodiments, computing resources 1304 can be configured to use the stored reference data stream data for training the new synthetic data model. In various embodiments, the training data can include both newly-received and stored data. When the synthetic data model is a Generative Adversarial Network, computing resources 1304 can be configured to train the new synthetic data model, in some embodiments, as described above with regard to FIGS. 8 and 9.”, thus wherein the generative synthetic data model is previously generated within a training time period corresponding to one or more training operations for the machine learning model is disclosed, because Walters teaches storing reference data stream data for subsequent use as training data and teaches that the stored reference data stream data may be received prior to beginning training of a new synthetic data model. Walters further teaches gathering data during a first time interval and using that gathered data to train a new synthetic model in a subsequent time interval, including use of stored reference data stream data for training the new synthetic data model. Thus, Walters teaches that the generative synthetic data model is generated during a training time period corresponding to training operations performed using historical and newly received training data)
Claims 2-5 and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Walters et al. (hereafter Walters) (US 10599957), in view of Fly et al. (hereinafter Fly) (US 10762444) and further in view of Chen et al. (hereinafter Chen), a non-patent literature reference titled “Developments in MLflow: A system to accelerate the machine learning lifecycle”).
Regarding Claim 2, Walters combined with Fly teaches all of the limitations of claim 1 as cited above and Walters further teaches:
wherein […] indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system. This information can be provided using interface 113 . The databases can include cloud-based databases (e.g., AMAZON WEB SERVICES S3 buckets) or on-premises databases. The information can include model information, such as the type and/or purpose of the model and any measures of classification error”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107”, thus wherein information indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model is disclosed, because Walters teaches that model storage stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage and can be provided from that storage in response to a request from dataset generator or model optimizer. Thus, model storage corresponds to the model registry, the stored synthetic data model and descriptive information correspond to information indicative of the generative synthetic data model, and the stored data models and model information correspond to storage in association with the machine learning model)
Walters combined with Fly does not explicitly teach […] a data model representation […].
However, Chen teaches […] a data model representation […] (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus a data model representation is disclosed, because Chen teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a representation of model-related information in a stored format. Thus, Chen’s bundled serialized model format and associated dependency information correspond to the data model representation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters and Fly with Chen, because Walters and Fly teach storing model related information, including information indicative of a generative synthetic data model, within registry like model storage associated with a machine learning model, while Chen teaches a Model Registry framework for managing the model deployment lifecycle and standardizing model handling through cataloguing, lineage tracking, and versioned model management. Incorporating Chen’s model representation approach into the Walters and Fly’s system would have allowed the stored generative synthetic model related information to be maintained in a more structured and reusable form within the model registry, thereby improving organization, lifecycle management, and reducing the likelihood of broken or inferior models being used in deployment (Chen, Page 1 – Section 1, “While traditional software has a well-defined set of product features to be built, ML development revolves around experimentation: ML developers constantly experiment with new datasets, models, software libraries, tuning parameters, etc. to optimize a metric such as model accuracy. Because model performance depends heavily on the input data and training process, reproducibility is paramount throughout ML development”, & Page 3 – Section 4.1, “To address these needs, we introduced the MLflow Model Registry: a collaborative hub for managing the model deployment lifecycle. In addition to providing cataloguing and lineage tracking capabilities, the Model Registry standardizes the model deployment workflow by enabling ML developers and deployment engineers to version their models and transition them through four logical stages: "Development," "Staging," "Production," and "Archived." Further, organizations can restrict access to particular stages on a per-user or per-role basis, and the Model Registry enables users to request stage transitions from colleagues. These stages structure the deployment process and provide a model review framework, guarding against common pitfalls, such as the deployment of broken or inferior models to production”)
Regarding Claim 3, Walters and Fly combined with Chen teaches all the limitations of claim 2 as cited above and Walters further teaches:
wherein the model registry comprises […] comprises a respective machine learning model and […] indicative of a respective generative synthetic data model corresponding to the respective machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103”, thus Walters teaches wherein the model registry comprises a respective machine learning model and information indicative of a respective generative synthetic data model corresponding to the respective machine learning model, because Walters teaches that model storage 109 stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage 109 and provided from that storage in response to a request. Thus, model storage 109 corresponds to the model registry, the stored data models correspond to the respective machine learning model, and the stored synthetic data model together with its associated stored information corresponds to information indicative of a respective generative synthetic data model corresponding to the respective machine learning model)
Chen further teaches […] a plurality of composite model data objects and each of the plurality of composite model data objects comprises […] a respective data model representation […] (Chen, Page 2 – Section 3.1, “MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles”, & Page 2 – Section 4.1, “In particular, several adopters in the IoT domain described use cases wherein a unique model is built for each independent device or entity. These training processes frequently produce tens of thousands or even hundreds of thousands of models. ML practitioners and deployment engineers must vet each of these models, map them to specific production applications, and deploy them systematically to guard against regressions and breakages”, & Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus Chen teaches a plurality of composite model data objects and each of the plurality of composite model data objects comprises a respective data model representation, because Chen teaches that the MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles, which corresponds to maintaining multiple model related objects in a registry, and further teaches use cases involving tens of thousands or hundreds of thousands of models, which corresponds to a plurality of such model related objects. Chen also teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a respective data model representation associated with each model object. Thus, Chen’s Model Registry corresponds to storage of a plurality of composite model data objects, the multiple catalogued models correspond to the plurality of such objects, and the bundled serialized model format and additional dependency information correspond to the respective data model representation)
Regarding Claim 4, Walters and Fly combined with Chen teaches all the limitations of claim 2 as cited above and Chen further teaches:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus wherein the data model representation comprises a serialized representation of the generative synthetic data model is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to representing the generative synthetic data model in serialized form. Thus, Chen’s serialized model format corresponds to the serialized representation of the generative synthetic data model)
Regarding Claim 5, Walters and Fly combined with Chen teaches all the limitations of claim 4 as cited above and Chen further teaches:
generating the generative synthetic data model by deserializing the data model representation (Chen, Page 2 – Section 2, “ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, & Page 3 – Section 4.1, “The Model Registry integrates with these continuous integration and deployment (CI/CD) tools by providing APIs for fetching models by version and stage that developers can incorporate into their existing CI/CD workflows”, thus generating the generative synthetic data model by deserializing the data model representation is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to storing the model in serialized form, and further teaches fetching models by version and stage for use in CI/CD workflows, which corresponds to retrieving and loading the serialized model for use. Thus, Chen’s serialized model format corresponds to the data model representation, and Chen’s fetching and loading of the serialized model corresponds to generating the generative synthetic data model by deserializing the data model representation)
Walters further teaches generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset (Walters, Par. [0041], “As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “In step 305 , in some embodiments, dataset generator 103 can generate synthetic data. Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109”, thus generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset is disclosed, because Walters teaches that dataset generator generates synthetic data using a data model without reliance on input data, where the data model is configured to generate data matching the statistical and content characteristics of a training dataset and may map from a random or pseudorandom vector to elements in the training data space. Walters further teaches that dataset generator generates synthetic data. Thus, Walters’ generated synthetic data corresponds to the plurality of evaluation samples, and the generated data matching the statistical and content characteristics of the training dataset corresponds to evaluation samples corresponding to the historical training dataset)
Regarding Claim 14, Walters combined with Fly teaches all of the limitations of claim 13 as cited above and Walters further teaches:
wherein […] indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system. This information can be provided using interface 113 . The databases can include cloud-based databases (e.g., AMAZON WEB SERVICES S3 buckets) or on-premises databases. The information can include model information, such as the type and/or purpose of the model and any measures of classification error”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103 . In various embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from model optimizer 107”, thus wherein information indicative of the generative synthetic data model is stored within a model registry in association with the machine learning model is disclosed, because Walters teaches that model storage stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage and can be provided from that storage in response to a request from dataset generator or model optimizer. Thus, model storage corresponds to the model registry, the stored synthetic data model and descriptive information correspond to information indicative of the generative synthetic data model, and the stored data models and model information correspond to storage in association with the machine learning model)
Walters combined with Fly does not explicitly teach […] a data model representation […].
However, Chen teaches […] a data model representation […] (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus a data model representation is disclosed, because Chen teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a representation of model-related information in a stored format. Thus, Chen’s bundled serialized model format and associated dependency information correspond to the data model representation)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Walters and Fly with Chen, because Walters and Fly teach storing model related information, including information indicative of a generative synthetic data model, within registry like model storage associated with a machine learning model, while Chen teaches a Model Registry framework for managing the model deployment lifecycle and standardizing model handling through cataloguing, lineage tracking, and versioned model management. Incorporating Chen’s model representation approach into the Walters and Fly’s system would have allowed the stored generative synthetic model related information to be maintained in a more structured and reusable form within the model registry, thereby improving organization, lifecycle management, and reducing the likelihood of broken or inferior models being used in deployment (Chen, Page 1 – Section 1, “While traditional software has a well-defined set of product features to be built, ML development revolves around experimentation: ML developers constantly experiment with new datasets, models, software libraries, tuning parameters, etc. to optimize a metric such as model accuracy. Because model performance depends heavily on the input data and training process, reproducibility is paramount throughout ML development”, & Page 3 – Section 4.1, “To address these needs, we introduced the MLflow Model Registry: a collaborative hub for managing the model deployment lifecycle. In addition to providing cataloguing and lineage tracking capabilities, the Model Registry standardizes the model deployment workflow by enabling ML developers and deployment engineers to version their models and transition them through four logical stages: "Development," "Staging," "Production," and "Archived." Further, organizations can restrict access to particular stages on a per-user or per-role basis, and the Model Registry enables users to request stage transitions from colleagues. These stages structure the deployment process and provide a model review framework, guarding against common pitfalls, such as the deployment of broken or inferior models to production”)
Regarding Claim 15, Walters and Fly combined with Chen teaches all the limitations of claim 14 as cited above and Walters further teaches:
wherein the model registry comprises […] comprises a respective machine learning model and […] indicative of a respective generative synthetic data model corresponding to the respective machine learning model (Walters, Par. [0044], “Model storage 109 can include one or more databases configured to store data models and descriptive information for the data models. Model storage 109 can be configured to provide information regarding available data models to a user or another system”, & Par. [0057], “Process 300 can then proceed to step 303 . In step 303 , dataset generator 103 can be configured to receive a synthetic data model from model storage 109 . In some embodiments, model storage 109 can be configured to provide the synthetic data model to dataset generator 103 in response to a request from dataset generator 103”, thus Walters teaches wherein the model registry comprises a respective machine learning model and information indicative of a respective generative synthetic data model corresponding to the respective machine learning model, because Walters teaches that model storage 109 stores data models and descriptive information for those data models, and further teaches that a synthetic data model is stored in model storage 109 and provided from that storage in response to a request. Thus, model storage 109 corresponds to the model registry, the stored data models correspond to the respective machine learning model, and the stored synthetic data model together with its associated stored information corresponds to information indicative of a respective generative synthetic data model corresponding to the respective machine learning model)
Chen further teaches […] a plurality of composite model data objects and each of the plurality of composite model data objects comprises […] a respective data model representation […] (Chen, Page 2 – Section 3.1, “MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles”, & Page 2 – Section 4.1, “In particular, several adopters in the IoT domain described use cases wherein a unique model is built for each independent device or entity. These training processes frequently produce tens of thousands or even hundreds of thousands of models. ML practitioners and deployment engineers must vet each of these models, map them to specific production applications, and deploy them systematically to guard against regressions and breakages”, & Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus Chen teaches a plurality of composite model data objects and each of the plurality of composite model data objects comprises a respective data model representation, because Chen teaches that the MLflow Model Registry is a collaborative hub for cataloguing models and managing their deployment lifecycles, which corresponds to maintaining multiple model related objects in a registry, and further teaches use cases involving tens of thousands or hundreds of thousands of models, which corresponds to a plurality of such model related objects. Chen also teaches that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to a respective data model representation associated with each model object. Thus, Chen’s Model Registry corresponds to storage of a plurality of composite model data objects, the multiple catalogued models correspond to the plurality of such objects, and the bundled serialized model format and additional dependency information correspond to the respective data model representation)
Regarding Claim 16, Walters and Fly combined with Chen teaches all the limitations of claim 14 as cited above and Chen further teaches:
wherein the data model representation comprises a serialized representation of the generative synthetic data model (Chen, Page 2 – Section 2, “similarly, MLflow Projects facilitates reproducible runs via a standard packaging format. ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, thus wherein the data model representation comprises a serialized representation of the generative synthetic data model is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to representing the generative synthetic data model in serialized form. Thus, Chen’s serialized model format corresponds to the serialized representation of the generative synthetic data model)
Regarding Claim 17, Walters and Fly combined with Chen teaches all the limitations of claim 16 as cited above and Chen further teaches:
generating the generative synthetic data model by deserializing the data model representation (Chen, Page 2 – Section 2, “ONNX [12] is a cross-library model serialization format; the MLflow Model format bundles serialized models with additional dependency information and introduces the concept of flavors, which enable users to load and evaluate models across multiple ML frameworks and levels of abstraction”, & Page 3 – Section 4.1, “The Model Registry integrates with these continuous integration and deployment (CI/CD) tools by providing APIs for fetching models by version and stage that developers can incorporate into their existing CI/CD workflows”, thus generating the generative synthetic data model by deserializing the data model representation is disclosed, because Chen teaches that ONNX is a model serialization format and that the MLflow Model format bundles serialized models with additional dependency information, which corresponds to storing the model in serialized form, and further teaches fetching models by version and stage for use in CI/CD workflows, which corresponds to retrieving and loading the serialized model for use. Thus, Chen’s serialized model format corresponds to the data model representation, and Chen’s fetching and loading of the serialized model corresponds to generating the generative synthetic data model by deserializing the data model representation)
Walters further teaches generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset (Walters, Par. [0041], “As an additional example, dataset generator 103 can be configured to generate synthetic data using a data model without reliance on input data. For example, the data model can be configured to generate data matching statistical and content characteristics of a training dataset. In some aspects, the data model can be configured to map from a random or pseudorandom vector to elements in the training data space”, & Par. [0058], “In step 305 , in some embodiments, dataset generator 103 can generate synthetic data. Dataset generator 103 can be configured, in some embodiments, to identify sensitive data items (e.g., account numbers, social security numbers, names, addresses, API keys, network or IP addresses, or the like) in the data received from model storage 109”, thus generating, using the generative synthetic data model, a plurality of evaluation samples corresponding to the historical training dataset is disclosed, because Walters teaches that dataset generator generates synthetic data using a data model without reliance on input data, where the data model is configured to generate data matching the statistical and content characteristics of a training dataset and may map from a random or pseudorandom vector to elements in the training data space. Walters further teaches that dataset generator generates synthetic data. Thus, Walters’ generated synthetic data corresponds to the plurality of evaluation samples, and the generated data matching the statistical and content characteristics of the training dataset corresponds to evaluation samples corresponding to the historical training dataset)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. US 20230104757 is pertinent because it teaches using a generative machine learning model, such as an autoencoder, variational autoencoder, recurrent neural network, or transformer, to generate embeddings from input interaction sequences, classify the embeddings to produce a machine-actor score, compare the score to thresholds, and take downstream actions such as blocking, quarantining, or permitting interactions. The reference further teaches unsupervised training on historical interaction data, frequent retraining with more recent data to address data drift and updating models using new input sequences to improve robustness against evolving invalid traffic. Because applicant likewise concerns machine-learning-based monitoring, performance evaluation, threshold-based response, retraining, and use of generative or unsupervised models with historical and current data, this reference is relevant to the claimed invention as it addresses drift-aware training, model updating, and threshold-based responsive actions in a machine-learning system, even though it was not relied upon as a primary basis for the rejection.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHLIET ADMASU whose telephone number is (571)272-0034. The examiner can normally be reached Mon-Fri, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.T.A./
Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights