DETAILED ACTION
This action is in response to the application filed 11/17/2023. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed inventions are directed to non-statutory subject matter without significantly more.
Claim 1
Step 1: The claim recites “A system”, and is therefore directed to the statutory category of machine
Step 2A Prong 1: The claim recites the following judicial exception(s)
detecting, in a production dataset, a data shift from a training dataset used to train a machine learning model: This can be performed as a mental process. One can merely identify differences between a production dataset and a training dataset.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the following additional element(s)
one or more processors and one or more non-transitory computer-readable media having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the one or more processors, causing operations comprising: This is mere instruction to execute the recited judicial exceptions with generic computer hardware (MPEP 2106.05(f)).
providing, to a first generative adversarial classifier, the training dataset to train the first generative adversarial classifier to classify whether data belongs to the training dataset: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
providing, to a second generative adversarial classifier, the production dataset to train the second generative adversarial classifier to classify whether data belongs to a subset of the production dataset that corresponds to the data shift: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
providing, to a generative adversarial generator, the production dataset to cause the generative adversarial generator to generate synthetic data for potential inclusion in an updated training dataset for training the machine learning model: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
providing, to the first generative adversarial classifier and the second generative adversarial classifier, the synthetic data to cause the first generative adversarial classifier and the second generative adversarial classifier to classify the synthetic data: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
in response to the first generative adversarial classifier indicating that the synthetic data belongs to the training dataset and the second generative adversarial classifier indicating that the synthetic data belongs to the subset corresponding to the data shift, excluding the synthetic data from the updated training dataset: This is mere instruction to exclude data based on a judicial exception in a generic manner (MPEP 2106.05(f)).
in connection with detecting the data shift in the production dataset, updating the machine learning model using the updated training dataset that excludes the synthetic data generated via the generative adversarial generator: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The following additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
one or more processors and one or more non-transitory computer-readable media having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the one or more processors, causing operations comprising: This is mere instruction to execute the recited judicial exceptions with generic computer hardware (MPEP 2106.05(f)).
providing, to a first generative adversarial classifier, the training dataset to train the first generative adversarial classifier to classify whether data belongs to the training dataset: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
providing, to a second generative adversarial classifier, the production dataset to train the second generative adversarial classifier to classify whether data belongs to a subset of the production dataset that corresponds to the data shift: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
providing, to a generative adversarial generator, the production dataset to cause the generative adversarial generator to generate synthetic data for potential inclusion in an updated training dataset for training the machine learning model: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
providing, to the first generative adversarial classifier and the second generative adversarial classifier, the synthetic data to cause the first generative adversarial classifier and the second generative adversarial classifier to classify the synthetic data: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
in response to the first generative adversarial classifier indicating that the synthetic data belongs to the training dataset and the second generative adversarial classifier indicating that the synthetic data belongs to the subset corresponding to the data shift, excluding the synthetic data from the updated training dataset: This is mere instruction to exclude data based on a judicial exception in a generic manner (MPEP 2106.05(f)).
in connection with detecting the data shift in the production dataset, updating the machine learning model using the updated training dataset that excludes the synthetic data generated via the generative adversarial generator: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 2
Step 1: The claim recites “A method”, and is therefore directed to the statutory category of process
Step 2A Prong 1: The claim recites the following judicial exception(s)
detecting, in a production dataset, a data shift from a training dataset used to train a machine learning model: This can be performed as a mental process. One can merely identify differences between a production dataset and a training dataset.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the following additional element(s)
providing, to a generative adversarial network comprising a first classifier and a second classifier, the training dataset to train the first classifier and the production dataset to train the second classifier: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
providing, to the generative adversarial network, synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the synthetic data: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
receiving, from the generative adversarial network, a first classification for the synthetic data from the first classifier and from the second classifier: This amounts to mere data output and is insignificant extra-solution activity (MPEP 2106.05(g)).
in response to receiving the first classification for the synthetic data from the first classifier and from the second classifier, excluding the synthetic data from an updated training dataset for updating the machine learning model: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The following additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
providing, to a generative adversarial network comprising a first classifier and a second classifier, the training dataset to train the first classifier and the production dataset to train the second classifier: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
providing, to the generative adversarial network, synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the synthetic data: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
receiving, from the generative adversarial network, a first classification for the synthetic data from the first classifier and from the second classifier: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
in response to receiving the first classification for the synthetic data from the first classifier and from the second classifier, excluding the synthetic data from an updated training dataset for updating the machine learning model: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 3
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
receiving, from the generative adversarial network, the first classification for the other synthetic data from the first classifier and a second classification for the other synthetic data from the second classifier: This amounts to mere data output and is insignificant extra-solution activity (MPEP 2106.05(g)).
in response to receiving the first classification from the first classifier and the second classification from the second classifier, excluding the other synthetic data from the updated training dataset: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
receiving, from the generative adversarial network, the first classification for the other synthetic data from the first classifier and a second classification for the other synthetic data from the second classifier: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
in response to receiving the first classification from the first classifier and the second classification from the second classifier, excluding the other synthetic data from the updated training dataset: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 4
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
receiving, from the generative adversarial network, a second classification for the other synthetic data from the first classifier and the first classification for the other synthetic data from the second classifier: This amounts to mere data output and is insignificant extra-solution activity (MPEP 2106.05(g)).
in response to receiving the second classification from the first classifier and the first classification from the second classifier, including the other synthetic data in the updated training dataset: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
receiving, from the generative adversarial network, a second classification for the other synthetic data from the first classifier and the first classification for the other synthetic data from the second classifier: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
in response to receiving the second classification from the first classifier and the first classification from the second classifier, including the other synthetic data in the updated training dataset: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 5
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
receiving, from the generative adversarial network, a second classification for the other synthetic data from the first classifier and from the second classifier: This amounts to mere data output and is insignificant extra-solution activity (MPEP 2106.05(g)).
in response to receiving the second classification from the first classifier and from the second classifier, excluding the other synthetic data from the updated training dataset
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
receiving, from the generative adversarial network, a second classification for the other synthetic data from the first classifier and from the second classifier: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
in response to receiving the second classification from the first classifier and from the second classifier, excluding the other synthetic data from the updated training dataset: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claim 6
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
providing, to the machine learning model, the updated training dataset to cause the machine learning model to update: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
providing, to the machine learning model, the updated training dataset to cause the machine learning model to update: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
Claim 7
Step 1: The claim recites a method, as in claim 6
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein causing the machine learning model to update comprises causing the machine learning model to update one or more weights used to generate predictions: This is a well-understood, routine, and conventional limitation in the field of machine learning, and is thus insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein causing the machine learning model to update comprises causing the machine learning model to update one or more weights used to generate predictions: This is a known technique in machine learning, as noted by Zhang (NOISY ECOLOGICAL DATA ENHANCEMENT VIA SPATIOTEMPORAL INTERPOLATION AND VARIANCE MAPPING, filed 6/13/2023, US 20240104432 A1): “In a traditional training technique, the predictions of the machine learning model would be compared directly with the input values (or at least a test set of values withheld from the training data) in order to evaluate the performance of the machine learning model and to adjust the weights of the parameters of the machine learning model (e.g., via gradient descent) in order to improve the performance” (Zhang, [0036])
Claim 8
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
further comprising providing, to a new machine learning model the updated training dataset to train the new machine learning model to generate predictions: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
further comprising providing, to a new machine learning model the updated training dataset to train the new machine learning model to generate predictions: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
Claim 9
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
providing, to a new machine learning model, the training dataset and the updated training dataset to train the new machine learning model to generate predictions, wherein the updated training dataset is weighted more heavily than the training dataset: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
providing, to a new machine learning model, the training dataset and the updated training dataset to train the new machine learning model to generate predictions, wherein the updated training dataset is weighted more heavily than the training dataset: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
Claim 10
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
wherein the first classification from a classifier indicates that the synthetic data belongs to a dataset on which the classifier is trained and a second classification from the classifier indicates that the synthetic data does not belong to the dataset on which the classifier is trained: Receiving classifications from classifiers still amounts to mere output and is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
wherein the first classification from a classifier indicates that the synthetic data belongs to a dataset on which the classifier is trained and a second classification from the classifier indicates that the synthetic data does not belong to the dataset on which the classifier is trained: Receiving classifications from classifiers is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
Claim 11
Step 1: The claim recites a method, as in claim 2
Step 2A Prong 1: The claim recites no further judicial exception(s)
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
providing, to the generative adversarial network, the production dataset to cause the generative adversarial network to derive the synthetic data from the production dataset: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
providing, to the generative adversarial network, the production dataset to cause the generative adversarial network to derive the synthetic data from the production dataset: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
Claim 12
Step 1: The claim recites “One or more non-transitory, computer-readable media” and is thus directed to the statutory category of article of manufacture.
Step 2A Prong 1: The claim recites the following further judicial exception(s)
detecting, in a second dataset, a data shift from a first dataset used to train a machine learning model: This can be performed as a mental process. One can merely identify differences between a second dataset and a first dataset.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through the further additional element(s)
One or more non-transitory, computer-readable media storing instructions that, when executed by one or more processors, cause operations: This is mere instruction to execute the recited judicial exception with generic computer hardware (MPEP 2106.05(f)).
providing, to a generative adversarial network comprising a first classifier and a second classifier, the first dataset to train the first classifier and the second dataset to train the second classifier: This amounts to mere data gathering and is insignificant extra-solution activity (MPEP 2106.05(g)).
obtaining, via the first classifier and the second classifier, a first classification of synthetic data derived from the second dataset: This amounts to mere data output and is insignificant extra-solution activity (MPEP 2106.05(g)).
in response to obtaining the first classification via the first classifier and the second classifier, excluding the synthetic data from an updated first dataset for updating the machine learning model: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Step 2B: The further additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s)
One or more non-transitory, computer-readable media storing instructions that, when executed by one or more processors, cause operations: This is mere instruction to execute the recited judicial exception with generic computer hardware (MPEP 2106.05(f)).
providing, to a generative adversarial network comprising a first classifier and a second classifier, the first dataset to train the first classifier and the second dataset to train the second classifier: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
obtaining, via the first classifier and the second classifier, a first classification of synthetic data derived from the second dataset: This is an instance of retrieving information from memory, a limitation known to be well-understood, routine, and conventional (MPEP 2106.05(d) II. iv.)
in response to obtaining the first classification via the first classifier and the second classifier, excluding the synthetic data from an updated first dataset for updating the machine learning model: This is mere instruction to update a machine learning model based on a judicial exception in a generic manner (MPEP 2106.05(f)).
Claims 13-20
Step 1: Claims 13-20 recite an article of manufacture, as in claim 12.
Step 2A Prong 1: Claims 13-20 recite the same judicial exception(s) as claims 3-9 & 11, respectively.
Step 2A Prong 2: The judicial exception(s) are not integrated into a practical application through any additional elements. The analysis of claims 13-20 at this step mirrors that of claims 3-9 & 11, respectively, with the exception that claims 13-20 are directed to “One or more non-transitory, computer-readable media storing instructions that, when executed by one or more processors, cause operations”, said operations mirroring those of claims 3-9 & 11. This is a mere instruction to apply the exceptions using generic computer equipment (MPEP 2106.05(f)).
Step 2B: The additional element(s) of the claim, taken alone or in combination, do not amount to significantly more than the recited judicial exception(s). The analysis of claims 13-20 at this step mirrors that of claims 3-9 & 11, with the exception that claims 13-20 are directed to “One or more non-transitory, computer-readable media storing instructions that, when executed by one or more processors, cause operations”, said operations mirroring those of claims 3-9 & 11. This is mere instruction to apply the exceptions using generic computer equipment (MPEP 2106.05(f)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-6, 8, 10-16, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Park (CLINICAL MODEL GENERALIZATION, published 1/6/2022, US 2022/0004881 A1) in view of Yin (METHOD AND SYSTEM FOR DIRECTED TRANSFER OF CROSS-DOMAIN DATA BASED ON HIGH-RESOLUTION REMOTE SENSING IMAGES, published 1/27/2022, US 20220028038 A1).
Regarding claim 1, Park discloses [a] system for reducing compute resource usage for model training related to data shifts by excluding synthetic data from training datasets that is not representative of the data shifts:
“At operation 102, the system analyzes clinical data characteristics relevant to a given AI model. In particular, in order to analyze the clinical data characteristics that are relevant to a given AI model, the system must be provided with an initial sample dataset (training dataset) which includes the relevant clinical data characteristics. The initial sample dataset is a set of genuine or authentic data provided to the system to facilitate training, validation, and testing of the AI model” (Park, [0030])
“In the illustrative example, at operation 104, the system compares the statistical distributions of the sample dataset (training dataset) with the target distributions and identifies, at operation 106, that the sample data from the North site underrepresents density A (0% of North site images have density A compared to 10% of the target distribution) and overrepresents density B (50% of North site images have density B compared to 40% of the target distribution)” (Park, [0042]). The shift in density A’s distribution between sample and target domains is a data shift.
“the system generates further training data to address the specific identified issues. In at least some embodiments of the present disclosure, generating further training data includes generating further synthetic data to address the identified issues” (Park, [0058])
“At operation 124, the system retrains the model using the further training data. In other words, the further training data generated at operation 122 is used to retrain the model to improve the performance of the model” (Park, [0060])
“At operation 506, the system applies a second discriminator to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed (exclud[ed]) from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic … at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (exclud[ed]) from the set of synthetic images.” (Park, [0081]). As one of ordinary skill in the art would know, reducing the number of data points in a training set will reduc[e] compute resource usage for model training with it.
“In the illustrative example, the system applies the second discriminator to the remaining synthetic images identified as having density A to verify that none of the images get classified as having density B. In this way, the system eliminates those synthetic images that fall into the overlap. Thus, the system generates only images having density A, to address the underrepresentation of density A images in the sample dataset, without incidentally also generating images that could also be classified as having density B, which would be counterproductive.” (Park, [0083]). Synthetic data not properly representative of the data shift of density A representation are excluded.
Park’s system comprising: one or more processors and one or more non-transitory computer-readable media having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the one or more processors, causing operations: “The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention” (Park, [0117]); “A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se” (Park, [0118])
Said operations comprising:
detecting, in a production dataset, a data shift from a training dataset used to train a machine learning model:
“Clinical models are artificial intelligence (AI) models that are generated by machine learning algorithms based on input datasets” (Park, [0019])
“When building the model (machine learning model) using the provided sample dataset (training dataset), knowing how the data distributions compare to distributions of clinical data (production dataset) characteristics that occur in the real world, also referred to as target distributions, enable generalization of the model and facilitate an understanding of how accurate and reliable the model can be when applied to new or future data. In other words, in order to generate a model that can be most accurately applied in the real world, the target distributions should be represented as closely as possible in the dataset used to train, validate, and test the AI model” (Park, [0035]).
“The method includes comparing a distribution of a clinical data characteristic of a genuine dataset (training dataset) with a target distribution (distribution of production dataset) of the clinical data characteristic to identify any categories of the clinical data characteristic that are underrepresented (data shift) in the genuine dataset. The method further includes generating an artificial test dataset based on the result of the comparison. The method further includes generating training data based on the artificial test dataset. The method further includes providing the training data to the AI model (machine learning model) to adapt the AI model” (Park, [0003])
“In the illustrative example, at operation 104, the system compares the statistical distributions of the sample dataset (training dataset) with the target distributions (distributions of the production dataset[s]) and identifies, at operation 106, that the sample data from the North site underrepresents density A (0% of North site images have density A compared to 10% of the target distribution) and overrepresents density B (50% of North site images have density B compared to 40% of the target distribution)” (Park, [0042]). The 10% shift in density A’s distribution between sample and target domains is a data shift.
providing, to a first generative adversarial classifier, the training dataset to train the first generative adversarial classifier to classify whether data belongs to the training dataset:
“At operation 502, the system generates synthetic images (e.g., artificial data) in the underrepresented category of the clinical data characteristic. More specifically, the system inputs a specific subset of the sample dataset (training dataset) into a generator of a multi-discriminator or restricted GAN (generative adversarial system). The subset of the sample dataset are all images in the underrepresented category of the clinical data characteristic. The generator then generates synthetic images in the same category based on these genuine images” (Park, [0077])
“At operation 506, the system applies a second discriminator (first generative adversarial classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset (training dataset) that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category” (Park, [0081])
“In the illustrative example, the system applies the second discriminator to the remaining synthetic images identified as having density A to verify that none of the images get classified as having density B. In this way, the system eliminates those synthetic images that fall into the overlap. Thus, the system generates only images having density A, to address the underrepresentation of density A images in the sample dataset, without incidentally also generating images that could also be classified as having density B, which would be counterproductive” (Park, [0082])
providing, to a second generative adversarial classifier, the production dataset to train the second generative adversarial classifier to classify whether data belongs to a subset of the production dataset that corresponds to the data shift:
“At operation 504, the system applies a first discriminator (second generative adversarial classifier) to the synthetic images (data) generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category (subset of the production dataset that corresponds to the data shift). In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed from the set of synthetic images” (Park, [0079])
“In the illustrative example, the system applies a first discriminator (second generative adversarial classifier) to the synthetic images having density A to verify that all of the synthetic images do, in fact, get classified as density A images. In at least some embodiments of the present disclosure, any images that do not get classified as density A images can be filtered out of or removed from the set of synthetic images” (Park, [0080])
providing, to a generative adversarial generator, the production dataset to cause the generative adversarial generator to generate synthetic data for potential inclusion in an updated training dataset for training the machine learning model:
“At operation 306, the system uses the generated synthetic data having clinical data characteristics in the second category to generate novel synthetic data having clinical data characteristics in the second category. … the novel synthetic data is new training data (data for an updated training dataset), which has distributions of other clinical data characteristics that match target distributions … In at least some embodiments of the present disclosure, the system can generate the novel synthetic data using a progressive GAN (generative adversarial generator)” (Park, [0071])
“In the illustrative example, the system inputs images identified as density A, which is underrepresented in the sample dataset relative to the target distribution, into the generator of the restricted GAN to generate synthetic images having density A” (Park, [0078])
“The method further includes generating training data based on the artificial test dataset. The method further includes providing the training data (updated training dataset) to the AI model (machine learning model) to adapt the AI model” (Park, [0003])
providing, to the first generative adversarial classifier and the second generative adversarial classifier, the synthetic data to cause the first generative adversarial classifier and the second generative adversarial classifier to classify the synthetic data:
“At operation 506, the system applies a second discriminator (first generative adversarial classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic” (Park, [0081])
“At operation 504, the system applies a first discriminator (second generative adversarial classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic” (Park, [0079])
in response to the first generative adversarial classifier indicating that the synthetic data belongs to the training dataset and the second generative adversarial classifier indicating that the synthetic data belongs to the subset corresponding to the data shift, excluding the synthetic data from the updated training dataset:
“At operation 506, the system applies a second discriminator (first generative adversarial classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset (training dataset) that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0081]).
“At operation 504, the system applies a first discriminator (second generative adversarial classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category (subset corresponding to the data shift). In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0079])
in connection with detecting the data shift in the production dataset, updating the machine learning model using the updated training dataset that excludes the synthetic data generated via the generative adversarial generator: “The method further includes generating training data (updated training dataset) based on the artificial test dataset. The method further includes providing the training data (updated training dataset) to the AI model (machine learning model) to adapt the AI model” (Park, [0003])
Park relates to generating a training data set corresponding to a data shift based on GAN-generated data and is analogous to the claimed invention.
While Park fails to disclose the further limitations of the claim, Yin discloses a system comprising:
providing, to a second generative adversarial classifier, the production dataset to train the second generative adversarial classifier to classify whether data belongs to a subset of the production dataset that corresponds to the data shift: “the step of inputting the ith training data sample in the training data sample subset into the image translation network model, and calculating a numerical value of the objective loss function of the image translation network model specifically includes:” (Yin, [0033]); “according to results of determining, by the target domain discriminator (second generative adversarial classifier), whether the target-domain image (production data) belongs to the target domain and whether the generated image (synthetic data) for the source domain belongs to the target domain (subset of the production dataset), calculating a value of an adversarial loss function for the
PNG
media_image1.png
93
937
media_image1.png
Greyscale
” (Yin, [0036]); “
D
Y
(
y
)
denotes a judgment whether true target-domain data y (production data) belongs to the target domain … G(x) denotes images which are generated after the source-domain images are subjected to forward mapping with the forward generator and conform to the probability distribution of the target domain” (Yin, [0011])
providing, to a generative adversarial generator, the production dataset to cause the generative adversarial generator to generate synthetic data for potential inclusion in an updated training dataset for training the machine learning model: “The training module can include: … input a target-domain image (production data) of the ith training data sample in the training data sample subset into the backward generator of the image translation network model, and carry out backward mapping, to obtain a generated image (synthetic data) corresponding to the target-domain image; and input the generated image corresponding to the target-domain image into the forward generator of the image translation network model, and carry out forward mapping, to obtain a reconstructed image corresponding to the target-domain image (synthetic data)” (Yin, [0142]).
Yin relates to multi-discriminator GANs for separating synthetic source and target domain data and is analogous to the claimed invention. Park teaches a system that uses a generator to produce synthetic data and a classifier to classify whether synthetic data points fall into a target domain. The claimed invention improves upon this system by providing target data to the data generator and using target data to train the synthetic data classifier. Yin teaches a method of training a generator to produce synthetic data by feeding it target data, and a method of training a classifier to discern whether synthetic data points fall within a target domain, using real target domain data, applicable to Park. A person of ordinary skill in the art would have recognized that training Park’s generator and classifier with target domain data would lead to the predictable result of the generator producing more realistic target domain data, and would improve the known device by increasing the accuracy of the synthetic data points and the classifiers filtering them (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Regarding claim 2, Park discloses [a] method comprising:
detecting, in a production dataset, a data shift from a training dataset used to train a machine learning model:
“Clinical models are artificial intelligence (AI) models that are generated by machine learning algorithms based on input datasets” (Park, [0019])
“At operation 102, the system analyzes clinical data characteristics relevant to a given AI model. In particular, in order to analyze the clinical data characteristics that are relevant to a given AI model, the system must be provided with an initial sample dataset (training dataset) which includes the relevant clinical data characteristics. The initial sample dataset is a set of genuine or authentic data provided to the system to facilitate training, validation, and testing of the AI model” (Park, [0030])
“When building the model (machine learning model) using the provided sample dataset (training dataset), knowing how the data distributions compare to distributions of clinical data (production dataset) characteristics that occur in the real world” (Park, [0035]).
“The method includes comparing a distribution of a clinical data characteristic of a genuine dataset (training dataset) with a target distribution (distribution of production dataset) of the clinical data characteristic to identify any categories of the clinical data characteristic that are underrepresented (data shift) in the genuine dataset” (Park, [0003])
“In the illustrative example, at operation 104, the system compares the statistical distributions of the sample dataset (training dataset) with the target distributions (distributions of the production dataset[s]) and identifies, at operation 106, that the sample data from the North site underrepresents density A (0% of North site images have density A compared to 10% of the target distribution) and overrepresents density B (50% of North site images have density B compared to 40% of the target distribution)” (Park, [0042]). The 10% shift in density A’s distribution between sample and target domains is a data shift.
providing, to a generative adversarial network comprising a first classifier and a second classifier, the training dataset to train the first classifier and the production dataset to train the second classifier:
“the system inputs a specific subset of the sample dataset into a generator of a multi-discriminator or restricted GAN (generative adversarial network)” (Park, [0077])
“The method further includes applying a first discriminator (second classifier) to the artificial data to select artificial data that is categorized in the underrepresented category. The method further includes applying a second discriminator (first classifier) to the artificial data to remove artificial data that is categorized in a second category of the clinical data characteristic” (Park, [0004])
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset (training dataset) that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category” (Park, [0081])
providing, to the generative adversarial network, synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the synthetic data:
“At operation 114, the system generates an artificial or synthetic test dataset based on the target distributions (production dataset distributions)” (Park, [0048])
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic” (Park, [0081])
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic” (Park, [0079])
receiving, from the generative adversarial network, a first classification for the synthetic data from the first classifier and from the second classifier; and in response to receiving the first classification for the synthetic data from the first classifier and from the second classifier, excluding the synthetic data from an updated training dataset for updating the machine learning model:
“the novel synthetic data is new training data (data for an updated training dataset), which has distributions of other clinical data characteristics that match target distributions” (Park, [0071])
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0079]). A first classification from this discriminator indicates that a synthetic image is in the underrepresented category corresponding to the production / target domain.
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0081]). A first classification from this discriminator indicates that a synthetic image is in a second category that overlaps with the underrepresented category.
Park relates to generating a training data set corresponding to a data shift based on GAN-generated data and is analogous to the claimed invention.
While Park fails to disclose the further limitations of the claim, Yin discloses providing, to a generative adversarial network comprising a first classifier and a second classifier, the training dataset to train the first classifier and the production dataset to train the second classifier:
“the step of inputting the ith training data sample in the training data sample subset into the image translation network model, and calculating a numerical value of the objective loss function of the image translation network model specifically includes:” (Yin, [0033])
“according to results of determining, by the source domain discriminator (first classifier), whether the source-domain image (training data) belongs to the source domain and whether the generated image (synthetic data) for the target domain belongs to the source domain, calculating a value of an adversarial loss function for the
PNG
media_image2.png
94
945
media_image2.png
Greyscale
“ (Yin, [0035])
“according to results of determining, by the target domain discriminator (second classifier), whether the target-domain image (production data) belongs to the target domain and whether the generated image (synthetic data) for the source domain belongs to the target domain, calculating a value of an adversarial loss function for the
PNG
media_image1.png
93
937
media_image1.png
Greyscale
” (Yin, [0036])
“
D
Y
(
y
)
denotes a judgment whether true target-domain data y (production data) belongs to the target domain,
D
X
(
x
)
denotes a judgment whether true source-domain data x (training data) belongs to the source domain, and G(x) denotes images which are generated after the source-domain images are subjected to forward mapping with the forward generator and conform to the probability distribution of the target domain; F(y) denotes images which are generated after the target-domain images are subjected to backward mapping with the backward generator and conform to the probability distribution of the source domain” (Yin, [0011])
Yin relates to multi-discriminator GANs for separating synthetic source and target domain data and is analogous to the claimed invention. Park teaches a method of using two classifiers to identify data in one domain and not in another. The claimed invention improves upon this method by training one classifier with the training dataset and the other with the production dataset. Yin teaches a method of training one domain-discerning classifier with a training (source) dataset and another with a production (target) dataset, applicable to Park. A person of ordinary skill in the art would have recognized that training Park’s classifiers with target and source domain datasets would lead to the predictable result of producing discriminators able to identify synthetic data points within a target domain and not a source domain, and would improve the known device by optimizing the classifiers through adversarial training (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Regarding claim 3, the rejection of claim 2 is incorporated. Park discloses a method, further comprising:
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data:
“At operation 114, the system generates an artificial or synthetic test dataset based on the target distributions (production dataset distributions). In at least some embodiments of the present disclosure, the synthetic test dataset includes distribution-specific images (other synthetic data). In such embodiments, the set of images that are generated to make up the synthetic test dataset has clinical data characteristics based on the target distributions and the gaps or discrepancies between the distributions of existing dataset and the target distributions” (Park, [0048])
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images (other synthetic data). In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic” (Park, [0081])
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images (other synthetic data) generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic” (Park, [0079])
receiving, from the generative adversarial network, the first classification for the other synthetic data from the first classifier and a second classification for the other synthetic data from the second classifier; and in response to receiving the first classification from the first classifier and the second classification from the second classifier, excluding the other synthetic data from the updated training dataset:
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0079]). A first classification from this discriminator indicates that a synthetic image is in the underrepresented category corresponding to the production / target domain. A second classification indicates that the synthetic image is not in the underrepresented category and should be excluded.
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0081]). A first classification from this discriminator indicates that a synthetic image is in a second category that overlaps with the underrepresented category and should be excluded. A second classification indicates that the synthetic image is not in the overlapping second category.
Regarding claim 4, the rejection of claim 2 is incorporated. Park discloses a method, further comprising:
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data:
“At operation 114, the system generates an artificial or synthetic test dataset based on the target distributions (production dataset distributions). In at least some embodiments of the present disclosure, the synthetic test dataset includes distribution-specific images (other synthetic data). In such embodiments, the set of images that are generated to make up the synthetic test dataset has clinical data characteristics based on the target distributions and the gaps or discrepancies between the distributions of existing dataset and the target distributions” (Park, [0048])
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images (other synthetic data). In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic” (Park, [0081])
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images (other synthetic data) generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic” (Park, [0079])
receiving, from the generative adversarial network, a second classification for the other synthetic data from the first classifier and the first classification for the other synthetic data from the second classifier; and in response to receiving the second classification from the first classifier and the first classification from the second classifier, including the other synthetic data in the updated training dataset:
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed (excluded) from the set of synthetic images” (Park, [0079]). A first classification from this discriminator indicates that a synthetic image is in the underrepresented category corresponding to the production / target domain and should be includ[ed]. A second classification indicates that the synthetic image is not in the underrepresented category.
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (excluded) from the set of synthetic images” (Park, [0081]). A first classification from this discriminator indicates that a synthetic image is in a second category that overlaps with the underrepresented category. A second classification indicates that the synthetic image is not in the overlapping second category and should be includ[ed].
Regarding claim 5, the rejection of claim 2 is incorporated. Park recites a method, further comprising:
providing, to the generative adversarial network, other synthetic data derived from the production dataset to cause the first classifier and the second classifier to classify the other synthetic data:
“At operation 114, the system generates an artificial or synthetic test dataset based on the target distributions (production dataset distributions). In at least some embodiments of the present disclosure, the synthetic test dataset includes distribution-specific images (other synthetic data). In such embodiments, the set of images that are generated to make up the synthetic test dataset has clinical data characteristics based on the target distributions and the gaps or discrepancies between the distributions of existing dataset and the target distributions” (Park, [0048])
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images (other synthetic data). In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic” (Park, [0081])
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images (other synthetic data) generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic” (Park, [0079])
receiving, from the generative adversarial network, a second classification for the other synthetic data from the first classifier and from the second classifier; and in response to receiving the second classification from the first classifier and from the second classifier, excluding the other synthetic data from the updated training dataset:
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0079]). A first classification from this discriminator indicates that a synthetic image is in the underrepresented category corresponding to the production / target domain. A second classification indicates that the synthetic image is not in the underrepresented category and should be exclud[ed].
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0081]). A first classification from this discriminator indicates that a synthetic image is in a second category that overlaps with the underrepresented category and should be exclud[ed]. A second classification indicates that the synthetic image is not in the overlapping second category.
Regarding claim 6, the rejection of claim 2 is incorporated. Park discloses a method, further comprising providing, to the machine learning model, the updated training dataset to cause the machine learning model to update:
“At operation 122, the system generates further training data (updated training dataset) to address the specific identified issues. In at least some embodiments of the present disclosure, generating further training data includes generating further synthetic data to address the identified issues” (Park, [0058])
“At operation 124, the system retrains (update[s]) the model (machine learning model) using the further training data. In other words, the further training data generated at operation 122 is used to retrain the model to improve the performance of the model” (Park, [0060])
Regarding claim 8, the rejection of claim 2 is incorporated. Park discloses a method, further comprising providing, to a new machine learning model, the updated training dataset to train the new machine learning model to generate predictions:
“At operation 122, the system generates further training data (updated training dataset) to address the specific identified issues. In at least some embodiments of the present disclosure, generating further training data includes generating further synthetic data to address the identified issues” (Park, [0058])
“At operation 124, the system retrains the model (machine learning model) using the further training data. In other words, the further training data generated at operation 122 is used to retrain the model to improve the performance of the model” (Park, [0060])
“The present disclosure relates generally to the field of computer aided diagnosis (CAD) systems, and more particularly to the use of CAD in clinical model validation” (Park, [0001]); “CAD systems are used in conjunction with artificial intelligence (AI) models to assist medical professionals in interpreting medical images. For example, CAD systems can be used to analyze digital images to identify patterns or anomalies. These identifications (predictions) can then be used in clinical models to generate an indication of a potential issue or disease in the patient. This indication can be used to inform the medical professional's decision making processes” (Park, [0002])
Regarding claim 10, the rejection of claim 2 is incorporated. Park further discloses a method, wherein the first classification from a classifier indicates that the synthetic data belongs to a dataset on which the classifier is trained and a second classification from the classifier indicates that the synthetic data does not belong to the dataset on which the classifier is trained:
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (excluded) from the set of synthetic images” (Park, [0081]). A first classification from this discriminator indicates that a synthetic data point is in a second category that from the discriminator training data. A second classification indicates that the synthetic data point is not in said second category.
Regarding claim 11, the rejection of claim 2 is incorporated. Yin discloses a method, further comprising providing, to the generative adversarial network, the production dataset to cause the generative adversarial network to derive the synthetic data from the production dataset: “The training module can include: … input a target-domain image (production data) of the ith training data sample in the training data sample subset into the backward generator of the image translation network model, and carry out backward mapping, to obtain a generated image (synthetic data) corresponding to the target-domain image; and input the generated image corresponding to the target-domain image into the forward generator of the image translation network model, and carry out forward mapping, to obtain a reconstructed image corresponding to the target-domain image (synthetic data)” (Yin, [0142]).
The existing combination teaches a system that uses a generator to produce synthetic data. The claimed invention improves upon this system by providing target data to the data generator. Yin teaches a method of training a generator to produce synthetic data by feeding it target data, applicable to Park. A person of ordinary skill in the art would have recognized that training Park’s generator with target domain data would lead to the predictable result of the generator producing more realistic target domain data, and would improve the known device by increasing the accuracy of the synthetic data points relative to real target domain data (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
Regarding claim 12, Park discloses [o]ne or more non-transitory, computer-readable media storing instructions that, when executed by one or more processors, cause operations: “The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention” (Park, [0117]); “A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se” (Park, [0118])
Park’s operations comprising:
detecting, in a second dataset, a data shift from a first dataset used to train a machine learning model:
“Clinical models are artificial intelligence (AI) models that are generated by machine learning algorithms based on input datasets” (Park, [0019])
“At operation 102, the system analyzes clinical data characteristics relevant to a given AI model. In particular, in order to analyze the clinical data characteristics that are relevant to a given AI model, the system must be provided with an initial sample dataset (first dataset) which includes the relevant clinical data characteristics. The initial sample dataset is a set of genuine or authentic data provided to the system to facilitate training, validation, and testing of the AI model” (Park, [0030])
“When building the model (machine learning model) using the provided sample dataset (first dataset), knowing how the data distributions compare to distributions of clinical data (second dataset) characteristics that occur in the real world” (Park, [0035]).
“The method includes comparing a distribution of a clinical data characteristic of a genuine dataset (first dataset) with a target distribution (distribution of second dataset) of the clinical data characteristic to identify any categories of the clinical data characteristic that are underrepresented (data shift) in the genuine dataset” (Park, [0003])
“In the illustrative example, at operation 104, the system compares the statistical distributions of the sample dataset (first dataset) with the target distributions (distributions of the second dataset[s]) and identifies, at operation 106, that the sample data from the North site underrepresents density A (0% of North site images have density A compared to 10% of the target distribution) and overrepresents density B (50% of North site images have density B compared to 40% of the target distribution)” (Park, [0042]). The 10% shift in density A’s distribution between sample and target domains is a data shift.
providing, to a generative adversarial network comprising a first classifier and a second classifier, the first dataset to train the first classifier and the second dataset to train the second classifier:
“the system inputs a specific subset of the sample dataset into a generator of a multi-discriminator or restricted GAN (generative adversarial network)” (Park, [0077])
“The method further includes applying a first discriminator (second classifier) to the artificial data to select artificial data that is categorized in the underrepresented category. The method further includes applying a second discriminator (first classifier) to the artificial data to remove artificial data that is categorized in a second category of the clinical data characteristic” (Park, [0004])
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset (first dataset) that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category” (Park, [0081])
obtaining, via the first classifier and the second classifier, a first classification of synthetic data derived from the second dataset:
“At operation 114, the system generates an artificial or synthetic test dataset based on the target distributions (second dataset distributions)” (Park, [0048])
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0079]). A first classification from this discriminator indicates that a synthetic image is in the underrepresented category corresponding to the second / target domain.
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0081]). A first classification from this discriminator indicates that a synthetic image is in a second category that overlaps with the underrepresented category.
in response to obtaining the first classification via the first classifier and the second classifier, excluding the synthetic data from an updated first dataset for updating the machine learning model:
“the novel synthetic data is new training data (data for an updated training dataset), which has distributions of other clinical data characteristics that match target distributions” (Park, [0071])
“At operation 504, the system applies a first discriminator (second classifier) to the synthetic images generated at operation 502. The first discriminator selects those images that are identified as being in the underrepresented category. In other words, the first discriminator checks to verify that all of the images generated at operation 502 are, in fact, identified as being in the underrepresented category of the clinical data characteristic. In at least some embodiments of the present disclosure, any images that do not get identified as being in the underrepresented category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0079]). A first classification from this discriminator indicates that a synthetic image is in the underrepresented category corresponding to the second / target domain.
“At operation 506, the system applies a second discriminator (first classifier) to the synthetic images. In embodiments where images that did not get identified as being in the underrepresented category were removed from the set of synthetic images, the system applies the second discriminator to the remaining synthetic images. The second discriminator checks that none of the synthetic images (or remaining synthetic images) are in a second category that overlaps with the underrepresented category of the clinical data characteristic. In order to apply the second discriminator, the system must also be supplied with a subset of the sample dataset that are images in this second, overlapping category in order to be trained to recognize images that will be identified as falling within the second category. In at least some embodiments of the present disclosure, any images that do get classified has being in the overlapping category can be filtered out of or removed (exclud[ed]) from the set of synthetic images” (Park, [0081]). A first classification from this discriminator indicates that a synthetic image is in a second category that overlaps with the underrepresented category.
Park relates to generating a training data set corresponding to a data shift based on GAN-generated data and is analogous to the claimed invention.
While Park fails to disclose the further limitations of the claim, Yin discloses providing, to a generative adversarial network comprising a first classifier and a second classifier, the first dataset to train the first classifier and the second dataset to train the second classifier:
“the step of inputting the ith training data sample in the training data sample subset into the image translation network model, and calculating a numerical value of the objective loss function of the image translation network model specifically includes:” (Yin, [0033])
“according to results of determining, by the source domain discriminator (first classifier), whether the source-domain image (first data) belongs to the source domain and whether the generated image (synthetic data) for the target domain belongs to the source domain, calculating a value of an adversarial loss function for the
PNG
media_image2.png
94
945
media_image2.png
Greyscale
“ (Yin, [0035])
“according to results of determining, by the target domain discriminator (second classifier), whether the target-domain image (second data) belongs to the target domain and whether the generated image (synthetic data) for the source domain belongs to the target domain, calculating a value of an adversarial loss function for the
PNG
media_image1.png
93
937
media_image1.png
Greyscale
” (Yin, [0036])
“
D
Y
(
y
)
denotes a judgment whether true target-domain data y (second data) belongs to the target domain,
D
X
(
x
)
denotes a judgment whether true source-domain data x (first data) belongs to the source domain, and G(x) denotes images which are generated after the source-domain images are subjected to forward mapping with the forward generator and conform to the probability distribution of the target domain; F(y) denotes images which are generated after the target-domain images are subjected to backward mapping with the backward generator and conform to the probability distribution of the source domain” (Yin, [0011])
Yin relates to multi-discriminator GANs for separating synthetic source and target domain data and is analogous to the claimed invention. Park teaches a method of using two classifiers to identify data in one domain and not in another. The claimed invention improves upon this method by training one classifier with the training dataset and the other with the production dataset. Yin teaches a method of training one domain-discerning classifier with a first (source) dataset and another with a second (target) dataset, applicable to Park. A person of ordinary skill in the art would have recognized that training Park’s classifiers with target and source domain datasets would lead to the predictable result of producing discriminators able to identify synthetic data points within a target domain and not a source domain, and would improve the known device by optimizing the classifiers through adversarial training (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
The analysis of claims 13-16, 18, and 20 mirrors that of claims 3-6, 8, and 11, respectively, with the exception that claims 13-16, 18, and 20 are directed to generic computer hardware which executes the methods of claims 3-6, 8, and 11, respectively. This generic hardware is taught by Park, as discussed regarding claim 12. Thus, claims 13-16, 18, and 20 are rejected under the same rationales used for claims 3-6, 8, and 11, respectively.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Park (CLINICAL MODEL GENERALIZATION, published 1/6/2022, US 2022/0004881 A1) in view of Yin (METHOD AND SYSTEM FOR DIRECTED TRANSFER OF CROSS-DOMAIN DATA BASED ON HIGH-RESOLUTION REMOTE SENSING IMAGES, published 1/27/2022, US 20220028038 A1), and further in view of Peterson (SELECTING A DISCONNECT FROM DIFFERENT TYPES OF CHANNEL DISCONNECTS BY TRAINING A MACHINE LEARNING MODULE, published 8/13/2020, US 20200257967 A1).
Regarding claim 7, the rejection of claim 6 is incorporated. While the aforementioned references fail to disclose the further limitations of the claim, Peterson discloses a method, wherein causing the machine learning model to update comprises causing the machine learning model to update one or more weights used to generate predictions: “In block 708, the machine learning module 242 is retrained (update[d]), by adjusting the initial weights, to improve the determination (prediction) of whether to select one of no disconnect, logical disconnect, or physical disconnect” (Peterson, [0064])
Peterson relates to updating machine learning models by updating weights used for predictions and is analogous to the claimed invention. The existing combination teaches a method of retraining a machine learning model using an updated training set. The claimed invention improves upon this method by updating predictive weights. Peterson teaches a method of retraining a machine learning model by updating predictive weights, applicable to the existing combination. A person of ordinary skill in the art would have recognized that retraining the model by updating weights would lead to the predictable result of optimizing the model for the updated training set, and would improve the known device by tailoring the model’s predictions for the target domain the updated training set attempts to capture (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
The analysis of claim 17 mirrors that of claim 7, with the exception that claim 17 is directed to generic computer hardware which executes the methods of claim 7. This generic hardware is taught by Park, as discussed regarding claim 12. Thus, claim 17 is rejected under the same rationales used for claim 7.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Park (CLINICAL MODEL GENERALIZATION, published 1/6/2022, US 2022/0004881 A1) in view of Yin (METHOD AND SYSTEM FOR DIRECTED TRANSFER OF CROSS-DOMAIN DATA BASED ON HIGH-RESOLUTION REMOTE SENSING IMAGES, published 1/27/2022, US 20220028038 A1) and further in view of Hoffman (EVALUATING RESPONSES TO OPEN-ENDED QUESTIONS, filed 11/10/2022, US 12,125,411 B1).
Regarding claim 9, the rejection of claim 2 is incorporated. Park discloses a method, further comprising providing, to a new machine learning model, the training dataset and the updated training dataset to train the new machine learning model to generate predictions, wherein the updated training dataset is weighted more heavily than the training dataset:
“At operation 122, the system generates further training data (updated training dataset) to address the specific identified issues. In at least some embodiments of the present disclosure, generating further training data includes generating further synthetic data to address the identified issues” (Park, [0058])
“At operation 124, the system retrains the model (machine learning model) using the further training data. In other words, the further training data generated at operation 122 is used to retrain the model to improve the performance of the model. Accordingly, following operation 124, the method 100 returns to operation 102 and begins again. In this way, the system can assess the accuracy and reliability of the model using the updated data generated through the method.” (Park, [0060])
“The present disclosure relates generally to the field of computer aided diagnosis (CAD) systems, and more particularly to the use of CAD in clinical model validation” (Park, [0001]); “CAD systems are used in conjunction with artificial intelligence (AI) models to assist medical professionals in interpreting medical images. For example, CAD systems can be used to analyze digital images to identify patterns or anomalies. These identifications (predictions) can then be used in clinical models to generate an indication of a potential issue or disease in the patient. This indication can be used to inform the medical professional's decision making processes” (Park, [0002])
PNG
media_image3.png
1159
818
media_image3.png
Greyscale
(Park, Figure 1). As made clear by Figure 1, the process of determining an updated training set and retraining the model is performed cyclically. An initial training dataset from one round precedes an updated training dataset that will be used for the next round of retraining.
While the aforementioned references fail to disclose the further limitations of the claim, Hoffman discloses a method, further comprising providing, to a new machine learning model, the training dataset and the updated training dataset to train the new machine learning model to generate predictions, wherein the updated training dataset is weighted more heavily than the training dataset:
“entailment model 136 may be a machine learning model that has been trained (or retrained) to compare two phrases for entailment, such as that described in connection with FIG. 2. Entailment model 136 outputs, in response to receiving learner answer 223A and model answer 213, evaluation 225A (prediction). Evaluation 225A characterizes the relationship between learner answer 223A and model answer 213 as one of: (1) entailment, (2) contradiction, or (3) neutrality” (Hoffman, column 18, paragraph 1)
“Existing training data (training dataset), such as Stanford Natural Language Inference corpus, could be used for initial training of entailment model 136” (Hoffman, column 13, paragraph 3)
“Since the new training data (updated training dataset) may be more representative of actual uses cases than a pre-existing or generic set of training data, it may be appropriate to cause the retraining process to more heavily weight the effect of the new training data in training entailment model 136 (machine learning model)” (Hoffman, column 19, paragraph 2)
Hoffman relates to retraining machine learning models and is analogous to the claimed invention. The existing combination teaches a method of retraining a machine learning model using updated training data. The claimed invention improves upon this method by more heavily weighting new training data. Hoffman teaches a method of more heavily weighting new training data for retraining, applicable to the existing combination. A person of ordinary skill in the art would have recognized that weighting newer training data more heavily would lead to the predictable result of making the model more ‘forgetful’ of weights learned during previous training cycles, and would improve the known device by making the least accurate training data from the earliest cycles have less impact than newer training data more accurate to the intended target domain (MPEP 2143 I. (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results).
The analysis of claim 19 mirrors that of claim 9, with the exception that claim 19 is directed to generic computer hardware which executes the methods of claim 9. This generic hardware is taught by Park, as discussed regarding claim 12. Thus, claim 19 is rejected under the same rationales used for claim 9.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Hu (Duplex Generative Adversarial Network for Unsupervised Domain Adaptation, published 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 1498-1507) discloses a method of using a source domain discriminator and target domain discriminator to generate domain invariant synthetic data
Hong (Active Surveillance And Learning For Machine Learning Model Authoring And Deployment, filed 7/31/2020, US 11954610 B2) discloses a method of training representations for target domain data using discriminators to filter source data
Lin (Defaulted Model Constructing Method Based On Data Migration, Device, Device And Medium, published 4/29/2022, CN 114418749 A) discloses a method of filtering source domain data not similar enough to a target domain
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaron P Gormley whose telephone number is (571)272-1372. The examiner can normally be reached Monday - Friday 12:00 PM - 8:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AG/Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights