DETAILED ACTION
This Non-Final communication is in response to application no. 17/564,02 filed on 12/28/2021. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 01/07/2026 has been entered which amends claims 1-2, 12, 20. Claims 1-20 are pending.
Response to Arguments
Applicant's arguments with respect to 35 U.S.C § 101 filed 01/07/2026 have been fully considered but they are not persuasive.
Applicant argues that the submitted claims do not fall within the abstract idea category of mental processes (pages 1-2 of applicant’s arguments). The examiner respectfully disagrees. The claimed invention recites “generating, as an output by a training data generator, a set of candidate training data”. Generating training data is something that could be performed in the human mind or with pen and paper. Adding “training data generator” (a generic computer component) does not make this any less of a mental process. Similarly the claim states “determining, by a second machine learning model, a set of importance factors based on both the inferences and the reference data”. Determining importance factors using inferences and reference data could be performed in the human mind. Adding “a second machine learning model” (a generic computer component) does not make this any less of a mental process.
Applicant argues that the claimed invention is an improvement to technology (pages 2-3 of applicant’s arguments). The applicant argues that identifying which portions of candidate training data were important to training improves the functioning of machine learning training systems. According to the MPEP 2106.05(a) the improvement cannot come from the judicial exception (abstract idea) alone. Here the improvement is directed to the abstract idea and the other limitations are additional elements.
Applicant argues that the claims recite additional elements that amount to significantly more. The examiner respectfully disagrees. The additional elements are directed to obtaining reference data, training a machine learning model and updating the data generator. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)). Training a machine learning model and updating the data generator are recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept (MPEP 2106.05(f)).
Thus, the 101 rejection is maintained.
Applicant’s arguments with respect to 35 U.S.C § 102 and 103 filed 01/07/2026 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
101 Subject Matter Eligibility analysis
Step 1: Claims 1-20 are within the four statutory categories (a process, machine, manufacture or composition of matter.) Claims 1-11 describe a process, and claims 12-20 describe a machine.
With respect to claim 1:
Step 2A Prong 1: The claim recites an abstract idea enumerated in the 2019 PEG.
generating a set of candidate training data; (This is an abstract idea of a “Mental Process”. The “generating” in its broadest reasonable interpretation is capable of being performable in the human mind or on paper. For example, without reciting any specifics of the generating process, a person can create generic information or a second dataset on paper while referencing a first dataset, and therefore, the generating is directed toward an abstract idea.)
determining a set of importance factors based on the set of inferences and a second machine learning model; (This is an abstract idea of a "Mental Process." The "determining" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The determination could be made manually by an individual.)
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
Additional elements:
obtain a set of reference data (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
As an output by a training data generator; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
training a first machine learning model based on the set of candidate training data, wherein the first machine learning model generates a set of inferences during the training based on the set of candidate training data; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
by a second machine learning model (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
updating the training data generator based on one or more distributions of properties determined based on the set of importance factors; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional element “obtain a set of reference data” adds insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
The additional elements “As an output by a training data generator”, “training a first machine learning model…”, “by a second machine learning model” and “updating the training data generator…” are recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept (MPEP 2106.05(f)).
When considered in combination, these additional elements represent insignificant extra-solution activity and mere instructions to apply an expectation, which do not provide an inventive concept.
Therefore, claim 1 is ineligible.
With respect to claim 2:
Step 2A Prong 1: claim 2, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2A Prong 2: The judicial exception is not integrated into a practical application.
One or more portions of the candidate training data included in the set of importance factors had at least a threshold effect in training the first machine learning model (this limitation amounts to adding insignificant extra-solution activity to the judicial exception).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional elements add insignificant extra-solution activity to the judicial exception and cannot provide an inventive concept. Storing and retrieving information in memory is directed to a well understood routine conventional activity of data transmission (MPEP 2106.05(d)(II)(iv)).
Therefore, claim 2 is ineligible.
With respect to claim 3:
Step 2A Prong 1: claim 3, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
the set of reference data set comprises a set of validation data; (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The additional element merely limits the judicial exception to a particular field of use and also cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 3 is ineligible.
With respect to claim 4:
Step 2A Prong 1: claim 4, which incorporates the rejection of claim 1, recites an abstract idea:
determining a first distribution of properties for the set of candidate training data and a second distribution of properties for the set of reference data; (This is an abstract idea of a "Mental Process." The "determining" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The determination could be made manually by an individual.)
Step 2a Prong 2: claim 4 does not recite any additional elements and thus cannot be integrated into a practical application.
Step 2B: claim 4 does not recite any additional elements.
Therefore, claim 4 is ineligible.
With respect to claim 5:
Step 2A Prong 1: claim 5, which incorporates the rejection of claim 4, recites an abstract idea:
determining whether the first distribution of properties for the set of candidate training data is within a threshold of the second distribution of properties for the set of validation data; (This is an abstract idea of a "Mental Process." The "determining" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The determination could be made manually by an individual.)
Step 2a Prong 2: claim 5 does not recite any additional elements.
Step 2B: claim 5 does not recite any additional elements.
Therefore, claim 5 is ineligible.
With respect to claim 6:
Step 2A Prong 1: claim 6, which incorporates the rejection of claim 5, does not recite an abstract idea.
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
the training data generator is updated in response to determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of validation data; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element is recited in a generic level and represents a generic computer component to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept.
Therefore, claim 6 is ineligible.
With respect to claim 7:
Step 2A Prong 1: claim 7, which incorporates the rejection of claim 5, recites the additional abstract ideas:
determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of validation data; (This is an abstract idea of a "Mental Process." The "determining" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The determination could be made manually by an individual.)
generating a second set of candidate training data; (This is an abstract idea of a “Mental Process”. The “generating” in its broadest reasonable interpretation is capable of being performable in the human mind or on paper. For example, without reciting any specifics of the generating process, a person can create generic information or a second dataset on paper while referencing a first dataset, and therefore, the generating is directed toward an abstract idea.)
determining a second set of importance factors; (This is an abstract idea of a "Mental Process." The "determining" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The determination could be made manually by an individual.)
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
based on a training data generator; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
training the first machine learning model based on the second set of candidate training data, wherein the first machine learning model generates a second set of inferences during the training based on the second set of candidate training data; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
based on the second set of inferences and the second machine learning model; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional elements “based on a training data generator”, “training the first machine learning model…” and “based on the second set of inferences and the second machine learning model” are recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept.
When considered in combination, these additional elements represent mere instructions to apply an exception and insignificant extra-solution activity, which do not provide an inventive concept.
Therefore, claim 7 is ineligible.
With respect to claim 8:
Step 2A Prong 1: claim 8, which incorporates the rejection of claim 5, recites an additional abstract idea:
determining that the first distribution of properties for the set of candidate training data is within a threshold of the second distribution of properties for the set of validation data; (This is an abstract idea of a "Mental Process." The "determining" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The determination could be made manually by an individual.)
generating training data; (This is an abstract idea of a “Mental Process”. The “generating” in its broadest reasonable interpretation is capable of being performable in the human mind or on paper. For example, without reciting any specifics of the generating process, a person can create generic information or a second dataset on paper while referencing a first dataset, and therefore, the generating is directed toward an abstract idea.)
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
based on the training data generator; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
the training data is provided to other machine learning models to train the other machine learning models; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional elements “based on a training data generator”, and “the training data is provided to other machine learning models …” are recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept.
When considered in combination, these additional elements represent mere instructions to apply an exception and insignificant extra-solution activity, which do not provide an inventive concept.
Therefore, claim 8 is ineligible.
With respect to claim 9:
Step 2A Prong 1: claim 9, which incorporates the rejection of claim 5, does not recite an abstract idea.
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
the first distribution of properties and the second distribution of properties are associated with labels for the set of validation data; (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
As explained above the additional element of “the first distribution…” merely limits the judicial exception to a particular field of use and also cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 9 is ineligible.
With respect to claim 10:
Step 2A Prong 1: claim 10, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
the training data generator comprises one or more of a generative adversarial network or a variational autoencoder; (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
As explained above the additional element of “the training data generator comprises …” merely limits the judicial exception to a particular field of use and also cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 10 is ineligible.
With respect to claim 11:
Step 2A Prong 1: claim 11, which incorporates the rejection of claim 1, does not recite an abstract idea.
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
the set of training data comprises synthetic training data; (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
As explained above the additional element of “the set of training data …” merely limits the judicial exception to a particular field of use and also cannot provide an inventive concept (MPEP 2106.05(h)).
Therefore, claim 11 is ineligible.
With respect to claim 12:
Step 2A Prong 1: the claim recites similar limitations as corresponding to claim 1. Therefore, the same analysis that was utilized under step 2A prong 1 for claim 1, as described above, is equally applicable to claim 12.
Step 2a Prong 2: The claim recites similar additional elements as corresponding to claim 1. Therefore, the same analysis that was utilizes under step 2A prong 1 for claim 1, as described above, is equally applicable to claim 12. The claim does recite more additional elements:
An apparatus, comprising: a memory configured to store data; and a processing device coupled to the memory, (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional elements “apparatus”, “memory” and “processing device” are recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept.
When considered in combination, these additional elements represent mere instructions to apply an exception and insignificant extra-solution activity, which do not provide an inventive concept.
Therefore, claim 12 is ineligible.
With respect to claim 13:
The claim recites similar limitations as corresponding to claims 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 13.
Therefore, claim 13 is ineligible.
With respect to claim 14:
The claim recites similar limitations as corresponding to claims 3. Therefore, the same subject matter analysis that was utilized for claim 3, as described above, is equally applicable to claim 14.
Therefore, claim 14 is ineligible.
With respect to claim 15:
The claim recites similar limitations as corresponding to claims 4. Therefore, the same subject matter analysis that was utilized for claim 4, as described above, is equally applicable to claim 15.
Therefore, claim 15 is ineligible.
With respect to claim 16:
The claim recites similar limitations as corresponding to claims 5. Therefore, the same subject matter analysis that was utilized for claim 5, as described above, is equally applicable to claim 16.
Therefore, claim 16 is ineligible.
With respect to claim 17:
The claim recites similar limitations as corresponding to claims 6. Therefore, the same subject matter analysis that was utilized for claim 6, as described above, is equally applicable to claim 17.
Therefore, claim 17 is ineligible.
With respect to claim 18:
The claim recites similar limitations as corresponding to claims 7. Therefore, the same subject matter analysis that was utilized for claim 7, as described above, is equally applicable to claim 18.
Therefore, claim 18 is ineligible.
With respect to claim 19:
The claim recites similar limitations as corresponding to claims 8. Therefore, the same subject matter analysis that was utilized for claim 8, as described above, is equally applicable to claim 19.
Therefore, claim 19 is ineligible.
With respect to claim 20:
Step 2A Prong 1: the claim recites similar limitations as corresponding to claim 1. Therefore, the same analysis that was utilized under step 2A prong 1 for claim 1, as described above, is equally applicable to claim 20.
Step 2a Prong 2: The claim recites similar additional elements as corresponding to claim 1. Therefore, the same analysis that was utilizes under step 2A prong 1 for claim 1, as described above, is equally applicable to claim 20. The claim does recite more additional elements:
A non-transitory computer-readable storage medium; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element “a non-transitory computer-readable storage medium” is recited in a generic level and it represents generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept.
Therefore, claim 20 is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 10-13, and 20 are rejected 35 U.S.C. 103 as being unpatentable by Dalli (Us 20220172050 A1) in view of Ghorbani (NPL: ‘Data Shapley: Equitable Valuation of Data for Machine Learning (Published 2019)).
Regarding claim 1, Dalli teaches the following:
A method, comprising (claim 1).
obtain a set of reference data (Fig. 16 real data samples 3004)
generating, as an output by a training data generator, a set of candidate training data based on a training data generator (Fig. 16: Generator 2710).
updating the training data generator based on one or more distributions of properties ([0150] “A model explanation may include coefficients θ of the explainable architecture x that may be utilized to explain the feature importance of the input features for a given observation” and [0164] “The detected bias, feature attributions and partition related explanations may be utilized as a feedback input 2600 to an explainable generator XG or a black-box generator G to tune and construct more realistic samples.”).
Dalli does not teach:
training a first machine learning model based on the set of candidate training data, including providing the set of candidate training data as training input to the first machine learning model, wherein the first machine learning model outputs a set of inferences during the training based on the set of candidate training data
Determining, by a second machine learning model, a set of importance factors based on the set of inferences and the set of reference data, wherein the set of importance factors identifies one or more portions of the candidate training data that were important to the training of the first machine learning model; and
However Ghorbani does:
training a first machine learning model based on the set of candidate training data, including providing the set of candidate training data as training input to the first machine learning model, wherein the first machine learning model outputs a set of inferences during the training based on the set of candidate training data (Section 4.2 Synthetic Data “For the first sets of data set we us a logistic regression model and for the second set we use both a logistic regression and a neural network with one hidden layer.”).
Determining, by a second machine learning model, a set of importance factors based on the set of inferences and the set of reference data, wherein the set of importance factors identifies one or more portions of the candidate training data that were important to the training of the first machine learning model; and (Section 1. Introduction page 2 “We propose data Shapley value, leveraging powerful results from game theory, to quantify the contribution of individual data points to a learning task.” And section 4. Experiments & Applications “Moreover we conduct two experiments showing that data points that are noisy or have label corruption will be assigned low Shapley value. Lastly we demonstrate that Shapley values can also give informative scores for groups of individuals. Taken together, these experiments suggest that, in addition to its equitable properties, data Shapley provides meaningful values to quantify the importance of data and can inform downstream analysis.”)
Dalli and Ghorbani are considered analogous art to the claimed invention because they are in the same field of endeavor being data/model evaluation. It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to combine the data generation system of Dalli the training and feedback of Ghorbani. One of ordinary skill in the art would have been motivated to do this as Ghorbani discusses using their system on synthetic data. (Section 4.2 Synthetic Data)
Regarding claim 2, Dalli in view of Ghorbani teaches claim 1 as outlined above. Ghorbani further teaches:
one or more portions of the candidate training data included in the set of importance factors had at least a threshold effect in training the first machine learning model (Section 1. Introduction page 2 “Moreover, our empirical studies demonstrate that data Shapley has several additional utilities: 1) it gives more insights into the importance of each data point than the common leave-one-out score; 2) it can identify outliers and corrupted data; 3) it can inform how to acquire future data to improve the predictor.”)
Regarding claim 10, Dalli in view of Ghorbani teaches claim 1 as outlined above. Dalli further teaches:
wherein the training data generator comprises one or more of a generative adversarial network or a variational autoencoder ([0020] “The GAN architecture may include a generator and discriminator, which are designed to compete against each other. The generator may generate samples that are then evaluated by the discriminator to determine whether the generated samples are from the training dataset or from the generator”).
Regarding claim 11, Dalli in view of Ghorbani teaches claim 1 as outlined above. Dalli further teaches:
wherein the set of training data comprises synthetic training data ([0101] “It may be contemplated that the data synthesis capabilities of white-box models can be used to generate additional synthetic training dataset samples that augment the training dataset”).
Regarding claim 12, Dalli teaches the following:
An apparatus, comprising: a memory configured to store data; a processing device coupled to the memory, the processing device configured to ([0054] “Further, many of the embodiments described herein are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It should be recognized by those skilled in the art that the various sequences of actions described herein can be performed by specific circuits (e.g., application specific integrated circuits (ASICs)) and/or by program instructions executed by at least one processor”).
obtain a set of reference data (Fig. 16 real data samples 3004)
generating, as an output by a training data generator, a set of candidate training data (Fig. 19: Generator 2900; [0180] “it is further contemplated that S [part of the generator] may be a symbolic model that generates simulated or synthetic data”).
updating the training data generator based on one or more distributions of properties determined based on the set of importance factors. ([0150] “A model explanation may include coefficients θ of the explainable architecture x that may be utilized to explain the feature importance of the input features for a given observation” and [0164] “The detected bias, feature attributions and partition related explanations may be utilized as a feedback input 2600 to an explainable generator XG or a black-box generator G to tune and construct more realistic samples.”).
Dalli does not teach:
train a first machine learning model based on the set of candidate training data, including providing the set of candidate training data as training input to the first machine learning model, wherein the first machine learning model outputs a set of inferences during the training based on the set of candidate training data
Determine, by a second machine learning model, a set of importance factors based on the set of inferences and the set of reference data, wherein the set of importance factors identifies one or more portions of the candidate training data that were important to the training of the first machine learning model; and
However Ghorbani does:
train a first machine learning model based on the set of candidate training data, including providing the set of candidate training data as training input to the first machine learning model, wherein the first machine learning model outputs a set of inferences during the training based on the set of candidate training data (Section 4.2 Synthetic Data “For the first sets of data set we us a logistic regression model and for the second set we use both a logistic regression and a neural network with one hidden layer.”).
Determine, by a second machine learning model, a set of importance factors based on the set of inferences and the set of reference data, wherein the set of importance factors identifies one or more portions of the candidate training data that were important to the training of the first machine learning model; and (Section 1. Introduction page 2 “We propose data Shapley value, leveraging powerful results from game theory, to quantify the contribution of individual data points to a learning task.” And section 4. Experiments & Applications “Moreover we conduct two experiments showing that data points that are noisy or have label corruption will be assigned low Shapley value. Lastly we demonstrate that Shapley values can also give informative scores for groups of individuals. Taken together, these experiments suggest that, in addition to its equitable properties, data Shapley provides meaningful values to quantify the importance of data and can inform downstream analysis.”)
Dalli and Ghorbani are considered analogous art to the claimed invention because they are in the same field of endeavor being data/model evaluation. It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to combine the data generation system of Dalli the training and feedback of Ghorbani. One of ordinary skill in the art would have been motivated to do this as Ghorbani discusses using their system on synthetic data. (Section 4.2 Synthetic Data)
Regarding claim 13, Dalli in view of Ghorbani teaches claim 12 as outlined above. Ghorbani further teaches:
the set of importance factors is determined by the second machine learning model; and the second machine learning model uses an importance function to determine the set of importance factors based on the set of inferences and the set of reference data (Section 3. Approximating Data Shapley describes how they decide what data is important specifically algorithm 1 and algorithm 2)
Regarding claim 20, Dalli teaches the following
A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to perform operations comprising ([0054] “Additionally, the sequence of actions described herein can be embodied entirely within any form of computer-readable storage medium such that execution of the sequence of actions enables the at least one processor to perform the functionality described herein”).
obtain a set of reference data (Fig. 16 real data samples 3004)
generating a set of candidate training data based on a training data generator (Fig. 19: Generator 2900; [0180] “it is further contemplated that S [part of the generator] may be a symbolic model that generates simulated or synthetic data”).
updating the training data generator based on one or more distributions of properties determined based on the set of importance factors. ([0150] “A model explanation may include coefficients θ of the explainable architecture x that may be utilized to explain the feature importance of the input features for a given observation” and [0164] “The detected bias, feature attributions and partition related explanations may be utilized as a feedback input 2600 to an explainable generator XG or a black-box generator G to tune and construct more realistic samples.”).
Dalli does not teach:
training a first machine learning model based on the set of candidate training data, including providing the set of candidate training data as training input to the first machine learning model, wherein the first machine learning model outputs a set of inferences during the training based on the set of candidate training data
Determining, by a second machine learning model, a set of importance factors based on the set of inferences and the set of reference data, wherein the set of importance factors identifies one or more portions of the candidate training data that were important to the training of the first machine learning model; and
However Ghorbani does:
training a first machine learning model based on the set of candidate training data, including providing the set of candidate training data as training input to the first machine learning model, wherein the first machine learning model outputs a set of inferences during the training based on the set of candidate training data (Section 4.2 Synthetic Data “For the first sets of data set we us a logistic regression model and for the second set we use both a logistic regression and a neural network with one hidden layer.”).
Determining, by a second machine learning model, a set of importance factors based on the set of inferences and the set of reference data, wherein the set of importance factors identifies one or more portions of the candidate training data that were important to the training of the first machine learning model; and (Section 1. Introduction page 2 “We propose data Shapley value, leveraging powerful results from game theory, to quantify the contribution of individual data points to a learning task.” And section 4. Experiments & Applications “Moreover we conduct two experiments showing that data points that are noisy or have label corruption will be assigned low Shapley value. Lastly we demonstrate that Shapley values can also give informative scores for groups of individuals. Taken together, these experiments suggest that, in addition to its equitable properties, data Shapley provides meaningful values to quantify the importance of data and can inform downstream analysis.”)
Dalli and Ghorbani are considered analogous art to the claimed invention because they are in the same field of endeavor being data/model evaluation. It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to combine the data generation system of Dalli the training and feedback of Ghorbani. One of ordinary skill in the art would have been motivated to do this as Ghorbani discusses using their system on synthetic data. (Section 4.2 Synthetic Data)
Claims 3-9 and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Dalli in view of Ghorbani and Maruta (US 20240078468 A1).
Regarding claim 3, Dalli in view of Ghorbani teaches claim 1 as outlined above. Maruta teaches:
the set of reference data set comprises a set of validation data (Fig. 6 & [0059] “dividing data [reference data] into training data D1_1 and validation data”).
Dalli, Ghorbani and Maruta are considered analogous art to the claimed invention because they are in the same field of endeavor. It would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention to combine the validation method of Maruta with the data generation system of Dalli. One of ordinary skill in the art would have been motivated to do this obtain accurate training data (Maruta: [0010] & [0011]).
Regarding claim 4, Dalli in view of Ghorbani teaches claim 1 as outlined above. Maruta teaches:
determining a first distribution of properties for the set of candidate training data and a second distribution of properties for the set of reference data ([0056] “the distribution of the training data D1_1 and the distribution of the test data D1_2”).
Regarding claim 5, Dalli in view of Ghorbani and Maruta teaches claim 4 as outlined above. Maruta further teaches:
determining whether the first distribution of properties for the set of candidate training data is within a threshold of the second distribution of properties for the set of reference data ([0056] “Next, the validation method recommending unit 12 determines whether or not the overlap between the distribution of the training data D1_1 and the distribution of the test data D1_2 in the column data C3 is equal to or more than a threshold value”).
Regarding claim 6, Dalli in view of Ghorbani and Maruta teaches claim 5 as outlined above. Maruta further teaches:
wherein the training data generator is updated in response to determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of reference data ([0159] “the distribution difference between the distribution of the training data and the distribution of the test data is less than the threshold value, the validation method recommending unit 12A recommends a validation method for extracting the training data and the validation data in such a manner that the localization relationship between the training data and the test data is maintained”).
Regarding claim 7, Dalli in view of Ghorbani and Maruta teaches claim 5 as outlined above. Dalli further teaches:
generating a second set of candidate training data based on a training data generator (Fig. 19: Generator 2900; [0180] “it is further contemplated that S [part of the generator] may be a symbolic model that generates simulated or synthetic data”).
Dalli does not teach:
training the first machine learning model based on the second set of candidate training data, wherein the first machine learning model generates a second set of inferences during the training based on the second set of candidate training data; and
determining a second set of importance factors based on the second set of inferences and the second machine learning model
determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of validation data:
Ghorbani teaches:
training the first machine learning model based on the second set of candidate training data, wherein the first machine learning model generates a second set of inferences during the training based on the second set of candidate training data; and (Section 4.2 Synthetic Data “For the first sets of data set we us a logistic regression model and for the second set we use both a logistic regression and a neural network with one hidden layer.”).
determining a second set of importance factors based on the second set of inferences and the second machine learning model (Section 1. Introduction page 2 “We propose data Shapley value, leveraging powerful results from game theory, to quantify the contribution of individual data points to a learning task.” And section 4. Experiments & Applications “Moreover we conduct two experiments showing that data points that are noisy or have label corruption will be assigned low Shapley value. Lastly we demonstrate that Shapley values can also give informative scores for groups of individuals. Taken together, these experiments suggest that, in addition to its equitable properties, data Shapley provides meaningful values to quantify the importance of data and can inform downstream analysis.” And there is discussions in section 3.2 on iterating the process.)
Ghorbani does not teach:
determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of validation data:
But Maruta does teach this ([0159] “the distribution difference between the distribution of the training data and the distribution of the test data is less than the threshold value, the validation method recommending unit 12A recommends a validation method for extracting the training data and the validation data in such a manner that the localization relationship between the training data and the test data is maintained”).
Regarding claim 8, Dalli in view of Ghorbani and Maruta teaches claim 5 as outlined above. Maruta further teaches:
determining that the first distribution of properties for the set of candidate training data is within a threshold of the second distribution of properties for the set of reference data, generating training data based on the training data generator, wherein the training data is provided to other machine learning models to train the other machine learning models ([0056] “the distribution of the training data D1_1 and the distribution of the test data D1_2 in the column data C3 is equal to or more than a threshold value (step ST1B). In the portion where the distributions overlap, the training data D1_1 and the test data D1_2 include the same data”).
Regarding claim 9, Dalli in view of Ghorbani and Maruta teaches claim 5 as outlined above. Maruta further teaches:
the first distribution of properties and the second distribution of properties are associated with labels for the set of reference data ([0059] “a concept diagram illustrating data divided into the training data D1_1 and the validation data in such a manner that the rates of the training labels are the same and the training labels do not overlap in the 3-fold cross validation”).
Regarding claim 14, Dalli in view of Ghorbani teaches claim 12 as outlined above. Maruta teaches:
the set of reference data set comprises a set of validation data (Fig. 6 & [0059] “dividing data [reference data] into training data D1_1 and validation data”).
Regarding claim 15, Dalli in view of Ghorbani teaches claim 12 as outlined above. Maruta teaches:
determining a first distribution of properties for the set of candidate training data and a second distribution of properties for the set of reference data ([0056] “the distribution of the training data D1_1 and the distribution of the test data D1_2”).
Regarding claim 16, Dalli in view of Ghorbani and Maruta teaches claim 15 as outlined above. Maruta further teaches:
determining whether the first distribution of properties for the set of candidate training data is within a threshold of the second distribution of properties for the set of reference data ([0056] “Next, the validation method recommending unit 12 determines whether or not the overlap between the distribution of the training data D1_1 and the distribution of the test data D1_2 in the column data C3 is equal to or more than a threshold value”).
Regarding claim 17, Dalli in view of Ghorbani and Maruta teaches claim 15 as outlined above. Maruta further teaches:
wherein the training data generator is updated in response to determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of valid reference data ([0159] “the distribution difference between the distribution of the training data and the distribution of the test data is less than the threshold value, the validation method recommending unit 12A recommends a validation method for extracting the training data and the validation data in such a manner that the localization relationship between the training data and the test data is maintained”).
Regarding claim 18, Dalli in view of Ghorbani and Maruta teaches claim 15 as outlined above. Dalli further teaches:
generating a second set of candidate training data based on a training data generator (Fig. 19: Generator 2900; [0180] “it is further contemplated that S [part of the generator] may be a symbolic model that generates simulated or synthetic data”).
training the first machine learning model based on the second set of candidate training data, wherein the first machine learning model generates a second set of inferences during the training based on the second set of candidate training data; and (Fig. 19 Simulator model S 2960 [equivalent to first machine learning model] and sample from simulator model 2910 [equivalent to inferences]).
determining a second set of importance factors based on the second set of inferences and the second machine learning model (Discriminator D (2720 from Fig. 19) [0180] “generated explanations [equivalent to importance factors] from the explainable discriminator [equivalent to second machine learning model]”. [0179] “explanations may be in multiple formats… numeric formats which may represent the importance of the input dimensions or the bias in the given input dimensions”).
Dalli does not teach:
training the first machine learning model based on the second set of candidate training data, wherein the first machine learning model generates a second set of inferences during the training based on the second set of candidate training data; and
determining a second set of importance factors based on the second set of inferences and the second machine learning model
determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of validation data:
Ghorbani teaches:
training the first machine learning model based on the second set of candidate training data, wherein the first machine learning model generates a second set of inferences during the training based on the second set of candidate training data; and (Section 4.2 Synthetic Data “For the first sets of data set we us a logistic regression model and for the second set we use both a logistic regression and a neural network with one hidden layer.”).
determining a second set of importance factors based on the second set of inferences and the second machine learning model (Section 1. Introduction page 2 “We propose data Shapley value, leveraging powerful results from game theory, to quantify the contribution of individual data points to a learning task.” And section 4. Experiments & Applications “Moreover we conduct two experiments showing that data points that are noisy or have label corruption will be assigned low Shapley value. Lastly we demonstrate that Shapley values can also give informative scores for groups of individuals. Taken together, these experiments suggest that, in addition to its equitable properties, data Shapley provides meaningful values to quantify the importance of data and can inform downstream analysis.” And there is discussions in section 3.2 on iterating the process.)
Ghorbani does not teach:
determining that the first distribution of properties for the set of candidate training data is not within a threshold of the second distribution of properties for the set of validation data:
But Maruta does teach this ([0159] “the distribution difference between the distribution of the training data and the distribution of the test data is less than the threshold value, the validation method recommending unit 12A recommends a validation method for extracting the training data and the validation data in such a manner that the localization relationship between the training data and the test data is maintained”).
Regarding claim 19, Dalli in view of Ghorbani and Maruta teaches claim 15 as outlined above. Maruta further teaches:
determining that the first distribution of properties for the set of candidate training data is within a threshold of the second distribution of properties for the set of reference data, generating training data based on the training data generator, wherein the training data is provided to other machine learning models to train the other machine learning models ([0056] “the distribution of the training data D1_1 and the distribution of the test data D1_2 in the column data C3 is equal to or more than a threshold value (step ST1B). In the portion where the distributions overlap, the training data D1_1 and the test data D1_2 include the same data”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL PATRICK GRUSZKA whose telephone number is (571)272-5259. The examiner can normally be reached M-F 9:00 AM - 6:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL GRUSZKA/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121