Detailed Action
This Office Action is in response to the remarks entered on 10/20/2025. Claim 2 is cancelled. Claims 1 and 3-20 currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 and 3-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
2A Prong 1:
applying at least one random modification to the base data sample, (a mental process of evaluation – modifying pattern by applying random modification can be done in the human mind with the aid of pen and paper) wherein the discriminator receives as input dataset pairs of datasets, the dataset pairs comprising each a prediction output of the generator based on a base data sample and the corresponding modified data sample, thereby optimizing a joint loss function for the generator and the discriminator, wherein the joint loss function is a Wassertein loss function; (mathematical concept – spec [0026-0027] a joint loss function measures a content loss between a base data sample and a modified data sample which directs to a mathematical concept)
predicting an output dataset for unknown data samples as input for the generator without the discriminator (mental process of judgment, as it merely discloses forecasting the output dataset based on unknown data which can be performed in human mind.).
generating,
2A Prong 2: This judicial exception is not integrated into a practical application.
A computer-implemented method for modifying patterns in datasets, the method using a generative adversarial network comprising a generator and a discriminator, (this recites method using GAN, which is merely apply it MPEP 2106.05(f).) the method comprising:
providing pairs of data samples, the pairs comprising each a base data sample and a modified data sample; (insignificant extra-solution activity MPEP 2106.05(g)(iii) of receiving data/data gathering.)
training, without the use of labeled or annotated data, the generator for building a model of the generator using an adversarial training method that inputs both samples of the pairs of data samples into the generator, (mere instructions to apply an exception using a computer MPEP 2106.05(f))
generating, using the trained generator (mere instructions to apply an exception using a computer MPEP 2106.05(f))
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
A computer-implemented method for modifying patterns in datasets, the method using a generative adversarial network comprising a generator and a discriminator, (this recites method using GAN, which is merely apply it MPEP 2106.05(f).) the method comprising:
providing pairs of data samples, the pairs comprising each a base data sample and a modified data sample; (was indicated as an insignificant extra-solution activity MPEP 2106.05(g)(iii), thus re-evaluated as a well understood, routine, and conventional activity MPEP 2106.05(d)(II)(iv) of gathering statistics.)
training, without the use of labeled or annotated data, the generator for building a model of the generator using an adversarial training method that inputs both samples of the pairs of data samples into the generator, (mere instructions to apply an exception using a computer MPEP 2106.05(f))
generating, using the trained generator (mere instructions to apply an exception using a computer MPEP 2106.05(f))
Regarding claim 3,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: further comprising training of different models for the generator network using the adversarial training method and using the pairs of data samples as input, (mere instructions to apply an exception MPEP 2106.05(f), as it discloses using a generator network.) wherein the modified data sample are modified according to a different aspect. (a field of use and technological environment MPEP 2106.05(h).)
2B: further comprising training of different models for the generator network using the adversarial training method and using the pairs of data samples as input, (mere instructions to apply an exception MPEP 2106.05(f), as it discloses using a generator network.) wherein the modified data sample are modified according to a different aspect. (a field of use and technological environment MPEP 2106.05(h).)
Regarding claim 4,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the generator is a neural network having as many output nodes as input nodes, and having less hidden layer nodes than the number of input nodes. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the number of nodes of the generator.)
2B: wherein the generator is a neural network having as many output nodes as input nodes, and having less hidden layer nodes than the number of input nodes. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the number of nodes of the generator.)
Regarding claim 5,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the discriminator is a neural network having as many input nodes as the generator has output nodes and having two output nodes. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the number of nodes of the discriminator.)
2B: wherein the discriminator is a neural network having as many input nodes as the generator has output nodes and having two output nodes. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the number of nodes of the discriminator.)
Regarding claim 6,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the discriminator is a PatchGAN. (a field of use and technological environment MPEP 2106.05(h).)
2B: wherein the discriminator is a PatchGAN. (a field of use and technological environment MPEP 2106.05(h).)
Regarding claim 7,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the joint loss function is a weighted combination of loss functions. (a field of use and technological environment MPEP 2106.05(h).)
2B: wherein the joint loss function is a weighted combination of loss functions. (a field of use and technological environment MPEP 2106.05(h).)
Regarding claim 8,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the loss function is related to content loss of the base data sample. (a field of use and technological environment MPEP 2106.05(h).)
and wherein the content loss is determined using a feature map of pre-trained neural network. (mere instructions to apply an exception MPEP 2106.05(f) as it discloses using a neural network.)
2B: wherein the loss function is related to content loss of the base data sample. (a field of use and technological environment MPEP 2106.05(h).)
and wherein the content loss is determined using a feature map of pre-trained neural network. (mere instructions to apply an exception MPEP 2106.05(f) as it discloses using a neural network.)
Regarding claim 9,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the randomly modified data sample comprises one or more of a randomly modified dashed line of the first data pattern to a continuous line, a randomly modified color of the first pattern, a randomly removed text pattern from the first pattern, and a randomly removed line pattern from the first pattern, wherein the random modification leaves at least one pattern of the plurality of patterns of the modified data sample unmodified. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the way of modifying the data sample.)
2B: wherein the randomly modified data sample comprises one or more of a randomly modified dashed line of the first data pattern to a continuous line, a randomly modified color of the first pattern, a randomly removed text pattern from the first pattern, and a randomly removed line pattern from the first pattern, wherein the random modification leaves at least one pattern of the plurality of patterns of the modified data sample unmodified. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the way of modifying the data sample.)
Regarding claim 10,
2A Prong 1: determining at least one pattern of the images to be modified (a mental process of evaluation, as it merely discloses determining which image to modify which can be performed in human mind.)
and relating one out of the set of images and a related image with the at least one pattern of the images defining one of the pairs comprising the base data sample and the modified data sample. (a mental process of evaluation, as it merely discloses figuring out the relationship between the set of images, which can be performed in human mind.)
2A Prong 2: wherein the providing pairs of data samples comprises: providing a set of images with patterns, (an insignificant extra-solution activity of mere data gathering MPEP 2106.05(g)(iii).)
randomly modifying the at least one pattern of the images using a random number generator, (an insignificant extra-solution activity of mere data gathering MPEP 2106.05(g)(iii).)
2B: wherein the providing pairs of data samples comprises: providing a set of images with patterns, (indicated as an insignificant extra-solution activity of mere data gathering MPEP 2106.05(g), thus re-evaluated as a well understood, routine, and conventional activity (MPEP 2106.05(d)(II)(iv) of gathering statistics.)
randomly modifying the at least one pattern of the images using a random number generator, (indicated as an insignificant extra-solution activity of mere data gathering MPEP 2106.05(g)(iii), thus re-evaluated as a well understood, routine, and conventional activity (MPEP 2106.05(d) according to Berkheimer evidence ([Hsu, US-20190252073-A1, 0110] The network architectures of the generator 560 and discriminator 570 are illustrated in FIG. 6 as generator 610 and discriminator 620. A traditional GAN is used to randomly generate arbitrary realism images from input noise data.)
Regarding claim 11,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the training of the generative adversarial network is terminated if a result of the joint loss function is smaller than a relative threshold value when comparing the result of the current iteration with a previous iteration. (mere instructions to apply an exception using a computer MPEP 2106.05(f), as it merely disclose stopping the training of the network based on the loss function result.)
2B: wherein the training of the generative adversarial network is terminated if a result of the joint loss function is smaller than a relative threshold value when comparing the result of the current iteration with a previous iteration. (mere instructions to apply an exception using a computer MPEP 2106.05(f), as it merely disclose stopping the training of the network based on the loss function result.)
Regarding claim 12,
2A Prong 1: Incorporates the rejection of claim 1.
2A Prong 2: wherein the training operation further comprises training the generator for building the model of the generator based on at least one of the base data samples, wherein the at least one of the base data samples lacks a corresponding modified data sample. (mere instructions to apply an exception using a computer MPEP 2106.05(f), as it merely recites training a neural network using the data sample.)
2B: wherein the training operation further comprises training the generator for building the model of the generator based on at least one of the base data samples, wherein the at least one of the base data samples lacks a corresponding modified data sample. (mere instructions to apply an exception using a computer MPEP 2106.05(f), as it merely recites training a neural network using the data sample.)
Regarding claim 13,
2A Prong 1:
predicting an output dataset for unknown data samples as input for the generator without the discriminator (a mental process of judgment, as it merely discloses forecasting the output dataset based on unknown data which can be performed in human mind.).
generating,
2A Prong 2: This judicial exception is not integrated into a practical application.
A machine-learning system for modifying patterns in datasets using a generative adversarial network, comprising a generator network system and a discriminator network system, the machine-learning system comprising a memory and at least one processor, coupled to said memory, and operative to perform operations comprising: (this recites method using GAN, which is merely apply it MPEP 2106.05(f).) the method comprising:
providing pairs of data samples, the pairs comprising each a base data sample and a modified data sample; (insignificant extra-solution activity MPEP 2106.05(g)(iii) of receiving data/data gathering.)
controlling a training of the generator network system for building a model of the generator network system using an adversarial training method that inputs both samples of the pairs of data samples into the generator, wherein the base data sample has a plurality of patterns, wherein the modified data sample has a first pattern of the plurality of patterns that is randomly modified in comparison to the base data sample, wherein the modified data sample has at least one pattern of the plurality of patterns that is unmodified in comparison to the base data sample, wherein the modified pattern is determined by applying at least one random modification to the base data sample, wherein the discriminator network system receives as input dataset pairs of datasets, the dataset pairs comprising each a prediction output of the generator based on a base data sample and the corresponding modified data sample, thereby optimizing a joint loss function for the generator and the discriminator, wherein the joint loss function is a Wassertein loss function; (directs to an insignificant extra-solution activity MPEP 2106.05(g) of training a Generative Adversarial Neural Network, as the limitation merely provides generic/standard GAN techniques for training the generator.)
generating, using the trained generator (mere instructions to apply an exception using a computer MPEP 2106.05(f))
2B: The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
A machine-learning system for modifying patterns in datasets using a generative adversarial network, comprising a generator network system and a discriminator network system, the machine-learning system comprising a memory and at least one processor, coupled to said memory, and operative to perform operations comprising: (this recites method using GAN, which is merely apply it MPEP 2106.05(f).)
providing pairs of data samples, the pairs comprising each a base data sample and a modified data sample; (was indicated as an insignificant extra-solution activity MPEP 2106.05(g), thus re-evaluated as a well understood, routine, and conventional activity MPEP 2106.05(d)(II)(iv) of gathering statistics.)
controlling a training of the generator network system for building a model of the generator network system using an adversarial training method that inputs both samples of the pairs of data samples into the generator, wherein the base data sample has a plurality of patterns, wherein the modified data sample has a first pattern of the plurality of patterns that is randomly modified in comparison to the base data sample, wherein the modified data sample has at least one pattern of the plurality of patterns that is unmodified in comparison to the base data sample, wherein the modified pattern is determined by applying at least one random modification to the base data sample, wherein the discriminator network system receives as input dataset pairs of datasets, the dataset pairs comprising each a prediction output of the generator based on a base data sample and the corresponding modified data sample, thereby optimizing a joint loss function for the generator and the discriminator, wherein the joint loss function is a Wassertein loss function; (is directed to an insignificant extra-solution activity MPEP 2106.05(g), thus re-evaluated as a well understood, routine, and conventional activity MPEP 2106.05(d) according to Berkheimer evidence ([Hsu, US-20190252073-A1, 0110] The network architectures of the generator 560 and discriminator 570 are illustrated in FIG. 6 as generator 610 and discriminator 620. A traditional GAN is used to randomly generate arbitrary realism images from input noise data.).
generating, using the trained generator (mere instructions to apply an exception using a computer MPEP 2106.05(f))
Regarding claim 14,
2A Prong 1: Incorporates the rejection of claim 13.
2A Prong 2: wherein the system trains different models for the generator network using the adversarial training method and using the pairs of data samples as input, (mere instructions to apply an exception MPEP 2106.05(f), as it discloses using a generator network.) wherein the modified data sample are modified according to a different aspect. (a field of use and technological environment MPEP 2106.05(h).)
2B: wherein the system trains different models for the generator network using the adversarial training method and using the pairs of data samples as input, (mere instructions to apply an exception MPEP 2106.05(f), as it discloses using a generator network.) wherein the modified data sample are modified according to a different aspect. (a field of use and technological environment MPEP 2106.05(h).)
Regarding claim 15,
2A Prong 1: Incorporates the rejection of claim 13.
2A Prong 2: wherein the generator network system is a neural network having as many output nodes as input nodes, and having less hidden layer nodes than the number of input nodes, or wherein the discriminator network system is a neural network having as many input nodes as the generator has output nodes and having two output nodes. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the number of nodes of the generator.)
2B: wherein the generator network system is a neural network having as many output nodes as input nodes, and having less hidden layer nodes than the number of input nodes, or wherein the discriminator network system is a neural network having as many input nodes as the generator has output nodes and having two output nodes. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the number of nodes of the generator.)
Regarding claim 16,
2A Prong 1: Incorporates the rejection of claim 13.
2A Prong 2: wherein the discriminator is a PatchGAN system. (a field of use and technological environment MPEP 2106.05(h) )
2B: wherein the discriminator is a PatchGAN system. (a field of use and technological environment MPEP 2106.05(h) )
Regarding claim 17,
2A Prong 1: Incorporates the rejection of claim 13.
2A Prong 2: wherein the loss function is related to content loss of the base data sample. (a field of use and technological environment MPEP 2106.05(h) )
and wherein the content loss is determined using a feature map of pre-trained neural network. (mere instructions to apply an exception MPEP 2106.05(f) as it discloses using a neural network.)
2B: wherein the loss function is related to content loss of the base data sample. (a field of use and technological environment MPEP 2106.05(h).)
and wherein the content loss is determined using a feature map of pre-trained neural network. (mere instructions to apply an exception MPEP 2106.05(f) as it discloses using a neural network.)
Regarding claim 18,
2A Prong 1: Incorporates the rejection of claim 13.
2A Prong 2: wherein the providing pairs of data samples comprises providing a set of images with patterns, determining at least one pattern of the images to be modified, randomly modifying the at least one pattern of the images using a random number generator, and relating one out of the set of images and a related image with the at least one pattern of the images defining one of the pairs comprising the base data sample and the modified data sample. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the way of modifying the data sample.)
2B: wherein the providing pairs of data samples comprises providing a set of images with patterns, determining at least one pattern of the images to be modified, randomly modifying the at least one pattern of the images using a random number generator, and relating one out of the set of images and a related image with the at least one pattern of the images defining one of the pairs comprising the base data sample and the modified data sample. (a field of use and technological environment MPEP 2106.05(h), as it merely defines the way of modifying the data sample.)
Regarding claim 19,
2A Prong 1: Incorporates the rejection of claim 13.
2A Prong 2: wherein the training operation further comprises training the generator for building the model of the generator based on at least one of the base data samples, wherein the at least one of the base data samples lacks a corresponding modified data sample. (mere instructions to apply an exception using a computer MPEP 2106.05(f), as it merely recites training a neural network using the data sample.)
2B: wherein the training operation further comprises training the generator for building the model of the generator based on at least one of the base data samples, wherein the at least one of the base data samples lacks a corresponding modified data sample. (mere instructions to apply an exception using a computer MPEP 2106.05(f), as it merely recites training a neural network using the data sample.)
Regarding claim 20,
2A Prong 1: Claim 20 is a computer program product claim having similar limitation to the claim 1 above. Therefore, the claim is rejected under the same rationale as claim 1 above.
2A Prong 2: A computer program product for modifying patterns in datasets using a generative adversarial network comprising a generator network system and a discriminator network system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, said program instructions being executable by one or more computing systems or controllers to cause said one or more computing systems to (this recites method using GAN, which is merely apply it MPEP 2106.05(f).).
2B: A computer program product for modifying patterns in datasets using a generative adversarial network comprising a generator network system and a discriminator network system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, said program instructions being executable by one or more computing systems or controllers to cause said one or more computing systems to (this recites method using GAN, which is merely apply it MPEP 2106.05(f).).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 4, 7, 11-13, and 19-20 are rejected under 35 U.S.C. 103 over Shrivastava (US 11475276 B1) in view of Ceccaldi (US 20190046068 A1) and further in view of Jiang (US 20200042613 A1).
Regarding claim 1, Shrivastava teaches a computer-implemented method for modifying patterns in datasets, the method using a generative adversarial network comprising a generator and a discriminator, the method comprising ([Shrivastava, column 12, line 61-66] “Therefore, to help make the refined synthetic images 430 more realistic, an adversarial cost term 450 may be added to the overall objective, according to some embodiments. For example, in one embodiments, a generative network G.sub.θ, such as generator 120, and a discriminative network D.sub.ϕ, such as discriminator 130, may both be learned.”
[Shrivastava, column 6, line 20 - 42] “Rather than generate images from scratch (e.g., the focus of most traditional generative models) a generative model, such as generator 120, may be coupled with a simulator or synthesizer, such as synthesizer 110 … Synthesizer 110 may generate synthetic data in any of various ways, according to different embodiments. Synthesizer 110 may be configured to generate synthetic data based on a set of training data including labeled real images in which the labels of the real images may be sufficient to generate corresponding synthetic images (i.e., synthetic images that look very similar to corresponding real images in shape, pose and/or appearance) …” The real images correspond to the base data, and the synthetic image corresponds to the modified data. The synthesizer modifies the real data. Generator generates refined synthetic image that goes into the discriminator based on the real image and the synthetic image pair):
providing pairs of data samples, the pairs comprising each a base data sample and a modified data sample ([Shrivastava, column 6, line 20 - 42] “Rather than generate images from scratch (e.g., the focus of most traditional generative models) a generative model, such as generator 120, may be coupled with a simulator or synthesizer, such as synthesizer 110, thereby allowing the refinement of synthetic data (e.g., to make them more realistic) by utilizing both a synthesizer and adversarial networks (e.g., generator 120 and discriminator 130), according to some embodiments. Synthesizer 110 may generate synthetic data in any of various ways, according to different embodiments. Synthesizer 110 may be configured to generate synthetic data based on a set of training data including labeled real images in which the labels of the real images may be sufficient to generate corresponding synthetic images (i.e., synthetic images that look very similar to corresponding real images in shape, pose and/or appearance). For example, in the case of depth images of a human hand, the position, shape and bone angles of the hand may be the same (or similar) for each pair of synthetic and real images. Additionally, in some embodiments, synthesizer 110 may be configured to generate an image from an input label vector s. Given this label vector, synthesizer 110 may generate a corresponding synthetic image, according to one embodiment.” The real images correspond to the base data, and the synthetic image corresponds to the modified data. The generator generates refined synthetic image that goes into the discriminator based on the real image and the synthetic image pair.
[Shrivastava, column 5, line 17-20] “discriminative network configured to receive both refined synthetic data from the generative network and real data (e.g., from a set of training data) and learns to distinguish between the two” teaches the discriminator receives both real (base data) and refined synthetic data (modified data) ),
training, , wherein the joint loss function is a Wassertein loss function; ([Shrivastava, column 6, line 20 - 42] The real images is the base data, and the synthetic image is the modified data. The generator generates refined synthetic image that goes into the discriminator based on the real image and the synthetic image pair which are further disclosed in [Shrivastava, col 12, line 48-54]. The generator is trained based on a set of training data including real images and synthetic images, which are further disclosed in [Shrivastava, col 12, line 48-54].
[Shrivastava, column 5, line 13-20] teaches the discriminator receives both real (base data) and refined synthetic data (modified data).
[Shrivastava, column 13, line 31-44] “As shown in block 510, a mini-batch of images including both real and synthetic may be loaded … The total loss for a mini-batch may, in some embodiments, be computed as the average of L.sub.D.sup.i over the mini-batch. After computing the current generative function G.sub.θ as in block 520, the discriminative parameters may be updated as in block 530 …” This total loss corresponds to the joint loss function for the generator and discriminator);
generating, using the trained generator, an additional text-based data sample. ([Shrivastava, col 7, line 30-34] The method is further evaluated based on dataset including text images with 2383 different fonts for the task of font recognition. The dataset contains both labeled synthetic data and partially labeled real-world data. [Shrivastava, column 6, line 20 - 42] The real images is the base data, and the synthetic image is the modified data. The generator generates refined synthetic image that goes into the discriminator based on the real image and the synthetic image pair which are further disclosed in [Shrivastava, col 12, line 48-54])
Shrivastava does not specifically disclose:
training without the use of labeled or annotated data;
a first pattern of the plurality of patterns that is randomly modified in comparison to the base data sample, wherein the random modification comprises a randomly removed pattern from the first pattern;
wherein the modified pattern is determined by applying at least one random modification to the base data sample;
predicting an output dataset for unknown data samples as input for the generator without the discriminator,
wherein the joint loss function is a Wasserstein loss function;
Ceccaldi teaches:
training without the use of labeled or annotated data; ([Ceccaldi, 0045] discloses training using the loss function from both the decoder network and an adversarial network trained to discriminate between features of the encoder network 401. The training process does not require any labeled input data)
predicting an output dataset for unknown data samples as input for the generator without the discriminator ([Ceccaldi, 0006] “A patient is scanned by the magnetic resonance imaging system to acquire magnetic resonance data. The magnetic resonance data is input to a machine learnt generator network trained to extract features from input magnetic resonance data and reconstruct domain independent images using the extracted features”, generative network reconstruct the original image (making prediction of original image) using extracted feature)
wherein the joint loss function is a Wasserstein loss function ([Ceccaldi, 0051] “During the training process, to avoid or limit one or more of the above referenced issues, the discriminator network 411 provides a gradient calculated using a Wasserstein distance. The Wasserstein distance is a measure of the differences between probability distributions”)
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava and Ceccaldi to use the method of predicting an output dataset for unknown data samples as input for the generator without the discriminator of Ceccaldi to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to improve the performance of generator, as testing out the generator independently is necessary to improve the performance of generator.
Shrivastava in view of Ceccaldi does not specifically disclose:
a first pattern of the plurality of patterns that is randomly modified in comparison to the base data sample, wherein the random modification comprises a randomly removed pattern from the first pattern; wherein the modified pattern is determined by applying at least one random modification to the base data sample;
Jiang teaches:
a first pattern of the plurality of patterns that is randomly modified in comparison to the base data sample, wherein the random modification comprises a randomly removed pattern from the first pattern ([Jiang, 0081] FIG. 8 includes noisy message generator 810 that processes a message and generates a noisy message by modifying the original message. Any appropriate techniques may be used to generate a noisy message from a message. The noisy message may be created, for example, by removing one or more words or characters, … or any combination of the foregoing. The modifications to create the noisy message may be performed randomly according to a probability distribution.
[Jiang, 0084] discloses a decoder neural network component 840 receiving the message feature vector for the noisy message and the original, non-noisy message (e.g., receive word embeddings of the non-noisy message or some other indication of the words of the non-noisy message). Noisy message corresponds to the randomly modified pattern, and non-noisy message corresponds to the base data sample. );
wherein the modified pattern is determined by applying at least one random modification to the base data sample ([Jiang, 0081] FIG. 8 includes noisy message generator 810 that processes a message and generates a noisy message by modifying the original message. Any appropriate techniques may be used to generate a noisy message from a message. The noisy message may be created, for example, by removing one or more words or characters, … or any combination of the foregoing. The modifications to create the noisy message may be performed randomly according to a probability distribution. );
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava, Ceccaldi, and Jiang to use the method of applying random modification, wherein the random modification comprises a randomly removed pattern from the first pattern of Jiang to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to improve the performance of discriminator by introducing more types of random deformation to the input data.
Regarding claim 13, Shrivastava teaches:
A machine-learning system for modifying patterns in datasets using a generative adversarial network, comprising a generator network system and a discriminator network system, the machine-learning system comprising a memory and at least one processor, coupled to said memory, and operative to perform operations ([Shrivastava, column 17, line 66 – column 18, line 12] “In at least some embodiments, a system and/or server that implements a portion or all of one or more of the methods and/or techniques described herein, including the techniques to refine synthetic images, to train and execute machine learning algorithms including neural network algorithms, and the like, may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media … computing device 9000 includes one or more processors 9010 coupled to a main memory 9020 (which may comprise both non-volatile and volatile memory modules, and may also be referred to as system memory) via an input/output (I/O) interface 9030.” This paragraph teaches the system comprising memory, processor, operatives.
[Shrivastava, column 12, line 61-66] “Therefore, to help make the refined synthetic images 430 more realistic, an adversarial cost term 450 may be added to the overall objective, according to some embodiments. For example, in one embodiments, a generative network G.sub.θ, such as generator 120, and a discriminative network D.sub.ϕ, such as discriminator 130, may both be learned.” [Shrivastava, column 6, line 20 - 42] “Rather than generate images from scratch (e.g., the focus of most traditional generative models) a generative model, such as generator 120, may be coupled with a simulator or synthesizer, such as synthesizer 110 … Synthesizer 110 may generate synthetic data in any of various ways, according to different embodiments. Synthesizer 110 may be configured to generate synthetic data based on a set of training data including labeled real images in which the labels of the real images may be sufficient to generate corresponding synthetic images (i.e., synthetic images that look very similar to corresponding real images in shape, pose and/or appearance) …” The real images correspond to the base data, and the synthetic image corresponds to the modified data. The synthesizer modifies the real data. Generator generates refined synthetic image that goes into the discriminator based on the real image and the synthetic image pair. This paragraph teaches the generator modifies (refines) the original synthetic image to refined synthetic image. The paragraphs teach the GAN system).
Claim 13 is a system claim having similar limitation to the method claim 1. Therefore, it is rejected with same rationale as claim 1 above.
Regarding claim 20, Shrivastava teaches:
A computer program product for modifying patterns in datasets using a generative adversarial network comprising a generator network system and a discriminator network system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, said program instructions being executable by one or more computing systems or controllers to cause said one or more computing systems ([Shrivastava, column 17, line 66 – column 18, line 12] “In at least some embodiments, a system and/or server that implements a portion or all of one or more of the methods and/or techniques described herein, including the techniques to refine synthetic images, to train and execute machine learning algorithms including neural network algorithms, and the like, may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media … computing device 9000 includes one or more processors 9010 coupled to a main memory 9020 (which may comprise both non-volatile and volatile memory modules, and may also be referred to as system memory) via an input/output (I/O) interface 9030.” This paragraph teaches the system comprising memory, processor, operatives.
[Shrivastava, column 12, line 61-66] “Therefore, to help make the refined synthetic images 430 more realistic, an adversarial cost term 450 may be added to the overall objective, according to some embodiments. For example, in one embodiments, a generative network G.sub.θ, such as generator 120, and a discriminative network D.sub.ϕ, such as discriminator 130, may both be learned.” [Shrivastava, column 6, line 20 - 42] “Rather than generate images from scratch (e.g., the focus of most traditional generative models) a generative model, such as generator 120, may be coupled with a simulator or synthesizer, such as synthesizer 110 … Synthesizer 110 may generate synthetic data in any of various ways, according to different embodiments. Synthesizer 110 may be configured to generate synthetic data based on a set of training data including labeled real images in which the labels of the real images may be sufficient to generate corresponding synthetic images (i.e., synthetic images that look very similar to corresponding real images in shape, pose and/or appearance) …” The real images correspond to the base data, and the synthetic image corresponds to the modified data. The synthesizer modifies the real data. Generator generates refined synthetic image that goes into the discriminator based on the real image and the synthetic image pair.” This paragraph teaches the generator modifies (refines) the original synthetic image to refined synthetic image).
Claim 20 is a computer program product claim having similar limitation to the method claim 1. Therefore, it is rejected with same rationale as claim 1 above.
Regarding claim 4, Shrivastava teaches the method of claim 1.
Shrivastava does not specifically disclose wherein the generator is a neural network having as many output nodes as input nodes, and having less hidden layer nodes than the number of input nodes.
Ceccaldi teaches wherein the generator is a neural network having as many output nodes as input nodes, and having less hidden layer nodes than the number of input nodes ([Ceccaldi, The first figure, 400 ‘Machine Learnt Network’] shows the equal size of data for the input and the output, [Ceccaldi, 0040] “Various units or layers may be used, such as convolutional, pooling (e.g., max pooling), deconvolutional, fully connected, or other types of layers. Within a unit or layer 435, any number of nodes is provided. For example, 100 nodes are provided. Later or subsequent units may have more, fewer, or the same number of nodes. In general, for convolution, subsequent units have more abstraction. For example, the first unit provides features from the image, such as one node or feature being a line found in the image. The next unit combines lines, so that one of the nodes is a corner. The next unit may combine features (e.g., the corner and length of lines) from a previous unit so that the node provides a shape indication. For transposed-convolution to reconstruct, the level of abstraction reverses. Each unit or layer 435 in the encoder 401 reduces the level of abstraction or compression while each unit or layer 435 in the decoder increases the level of abstraction”).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava and Ceccaldi to use the generator having as many output nodes as input nodes, and having less hidden layer nodes than the number of input nodes of Ceccaldi to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to generate a value that does not exist in the original dataset, as compressing the input data using less number of hidden nodes deletes some information in the input data, and then decompressing the data adds new information that wasn’t in the original input data.
Regarding claim 7, Shrivastava teaches the method of claim 1.
Shrivastava does not specifically disclose wherein the joint loss function is a weighted combination of loss functions.
Ceccaldi teaches wherein the joint loss function is a weighted combination of loss functions ([Ceccaldi, Claim 19] “The system of claim 18, wherein the generator network is trained using a loss function that is calculated as a combination of a first value from a first loss function provided by the decoder and a second value from a second loss function provided by an adversarial learnt network trained to classify concatenated features from the compact representation as either from a first domain or a second domain”).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava and Ceccaldi to use the joint loss function which is a weighted combination of loss functions of Ceccaldi to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to improve the accuracy of the loss function by combining the two different ways of calculating probability distributions ([Ceccaldi, 0069]).
Regarding claim 11, Shrivastava teaches wherein the training of the generative adversarial network is terminated ([Shrivastava, col 14, line 1-3] The method of FIG. 6 may repeat until all the discriminative network updates for this currently step have been performed, as in decision block 630)
Shrivastava does not specifically disclose training is terminated if a result of the joint loss function is smaller than a relative threshold value when comparing the result of the current iteration with a previous iteration.
Ceccaldi teaches training is terminated if a result of the joint loss function is smaller than a relative threshold value when comparing the result of the current iteration with a previous iteration ([Ceccaldi, Claim 10] calculating, by the discriminator network, a second value based on a second loss function for the classification; adjusting the encoder network as a function of the first value and second value; and repeating sequentially inputting, inputting, calculating, inputting, calculating, and adjusting until a training loss converges).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava and Ceccaldi to use terminating the training process if a result of the joint loss function is smaller than a relative threshold value when comparing the
result of the current iteration with a previous iteration of Ceccaldi to implement the autonomous data
modification system of Shrivastava. The suggestion and/or motivation for doing so is to improve the
efficiency of the system, as repeating the training process even after the loss function has converged
does not improve the performance of the trained machine learning model and waste computation
resources.
Regarding claim 12, Shrivastava teaches:
wherein the training operation further comprises training the generator for building the model of the generator based on at least one of the base data samples, wherein the at least one of the base data samples lacks a corresponding modified data sample ([Shrivastava, col 12, line 33-41; and Fig. 4] the real image 440 inputs to the discriminator to train both the generator and discriminator does not have any corresponding input synthetic image. Additionally, [Shrivastava, col 16, line 54 – col 17, line 5] indicates that the adversarial real data that is used in an adversarial net refinement process may not have to include actual examples of the real data. Instead, similar real (e.g., refined) data that can provide refinement information can be used. The similar real data that is similar to the y_i does not have any corresponding modified data sample.).
Claim 19 is a system claim having similar limitation to the method claim 12. Therefore, it is rejected with same rationale as claim 12 above.
Claim 3, 6, 8-9, 10, 14 and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Shrivastava (US 11475276 B1) in view of Ceccaldi (US 20190046068 A1) in view of Jiang (US 20200042613 A1) and further in view of Yi (Yi et al, 2018, “DualGAN: Unsupervised Dual Learning for Image-to-Image Translation”)
Regarding claim 3, Shrivastava teaches:
further comprising training of different models for the generator network using the adversarial training method and using the pairs of data samples as input, wherein the modified data sample are modified ([Shrivastava, column 12, line 61-66] “Therefore, to help make the refined synthetic images 430 more realistic, an adversarial cost term 450 may be added to the overall objective, according to some embodiments. For example, in one embodiments, a generative network G.sub.θ, such as generator 120, and a discriminative network D.sub.ϕ, such as discriminator 130, may both be learned.”
[Shrivastava, column 12, line 48-54] “Given pairs of synthetic and real images, generator 120 may be configured to minimize the l.sub.1 or l.sub.2 norm of the image difference 460 between the original synthetic image and the refined synthetic image. Image difference 460 may be based on a comparison (e.g., mathematically) of the original synthetic image and the refined synthetic image, in some embodiments.” This paragraph teaches the generator modifies (refines) the original synthetic image to refined synthetic image. The different models for generator network is taught by another reference Yi).
Shrivastava in view of Ceccaldi and further in view of Jiang does not specifically disclose:
wherein the modified data sample are modified according to a different aspect.
Yi teaches:
wherein the modified data sample are modified according to a different aspect ([Yi, page 3, Figure 1] is showing the Generator and the Discriminator receiving a pair of inputs (fake, real), and using different Generators GA and GB.
[Yi, page 3, left column, second paragraph, line 8-10] “The discriminator DA is trained with v as positive samples and GA(u; z) as negative examples, whereas DB takes u as positive and GB(v; z0) as negative”, are showing the invention of Yi receiving a pair of inputs in the generator and the discriminator. [Yi, page 6, Figure 4; Figure 6] “Figure 4: Photo!sketch translation for faces. Results of DualGAN are generally sharper than those from cGAN, even though the former was trained using unpaired data, whereas the latter makes use of image correspondence” shows the process of modifying the data sample according to the various aspects, for example, photo->sketch, or Chinese painting->oil painting).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava, Ceccaldi, Jiang, and Yi to use the method of modified data sample are modified according to a different aspect of Yi to implement the autonomous data modification system of Shrivastava, Ceccaldi, and Jiang. The suggestion and/or motivation for doing so is to enable the invention to handle a wider variety of types of data.
Claim 14 is a system claim having similar limitation to the method claim 3. Therefore, it is rejected with same rationale as claim 3 above.
Regarding claim 6, Shrivastava teaches:
utilizing the discriminator ([Shrivastava, column 12, line 61-66] “Therefore, to help make the refined synthetic images 430 more realistic, an adversarial cost term 450 may be added to the overall objective, according to some embodiments. For example, in one embodiments, a generative network G.sub.θ, such as generator 120, and a discriminative network D.sub.ϕ, such as discriminator 130, may both be learned.”).
Shrivastava in view of Ceccaldi in view of Jiang does not specifically disclose:
wherein the discriminator is a PatchGAN.
Yi teaches:
wherein the discriminator is a PatchGAN ([Yi, page 3, right column, 3.2 Network Configuration, second paragraph] “For discriminators, we employ the Markovian PatchGAN architecture as explored in [8], which assumes independence between pixels distanced beyond a specific patch size and models images only at the patch level rather than over the full image. Such a configuration is effective in capturing local high-frequency features such as texture and style, but less so in modeling global distributions”).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava, Ceccaldi, Jiang, and Yi to use the PatchGAN of Yi to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to improve the performance of the system, as PatchGAN is effective in capturing local high-frequency features such as texture and style ([Yi, page 3, right column, 3.2 Network Configuration, second paragraph]).
Claim 16 is a system claim having similar limitation to the method claim 6. Therefore, it is rejected with same rationale as claim 6 above.
Regarding claim 8, Shrivastava in view of Ceccaldi in view of Jiang teaches:
The method of claim 1.
Shrivastava in view of Ceccaldi in view of Jiang does not specifically disclose:
wherein the loss function is related to content loss of the base data sample and wherein the content loss is determined using a feature map of a pre-trained neural network.
Yi teaches:
wherein the loss function is related to content loss of the base data sample and wherein the content loss is determined using a feature map of a pre-trained neural network ([Yi, page 5, the last line of the left column – 4th line of the right column] “In all these tasks, cGAN was trained with labeled (i.e., paired) data, where we ran the model and code provided in [4] and chose the optimal loss function for each task: L1 loss for facade!label and L1 + cGAN loss for the other tasks (see [4] for more details). In contrast, DualGAN and GAN were trained in an unsupervised way, i.e., we decouple the image pairs and then reshuffle the data.”, Yi calculates the loss for each translation of the images, using DualGAN and GAN were pre-trained in unsupervised way).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava, Ceccaldi, Jiang, and Yi to use the method of wherein the loss function is related to content loss of the base data sample and wherein the content loss is determined using a feature map of a pre-trained neural network of Yi to implement the autonomous data modification system of Shrivastava, Ceccaldi, and Jiang. The suggestion and/or motivation for doing so is to improve the performance of the system, as calculating loss of each feature map of a neural network helps the system to track the loss of information during the processing of the data ([Yi, page 5, the last line of the left column – 4th line of the right column]).
Claim 17 is a system claim having similar limitation to the method claim 8. Therefore, it is rejected with same rationale as claim 8 above.
Regarding claim 9, Shrivastava in view of Ceccaldi in view of Jiang teaches:
wherein the randomly modified data sample comprises one or more of [Jiang, 0081] FIG. 8 includes noisy message generator 810 that processes a message and generates a noisy message by modifying the original message. Any appropriate techniques may be used to generate a noisy message from a message. The noisy message may be created, for example, by removing one or more words or characters, … or any combination of the foregoing. The modifications to create the noisy message may be performed randomly according to a probability distribution. ).
Shrivastava in view of Ceccaldi in view of Jiang does not specifically disclose:
wherein the randomly modified data sample comprises one or more of a randomly modified dashed line of the first data pattern to a continuous line, a randomly modified color of the first pattern, and a randomly removed line pattern from the first pattern.
Yi teaches:
wherein the randomly modified data sample comprises one or more of a randomly modified dashed line of the first data pattern to a continuous line, a randomly modified color of the first pattern, and a randomly removed line pattern from the first pattern ([Yi, page 3, left col, line 5-7] “As shown in Fig. 1, image u 2 U is translated to domain V using GA. How well the translation GA(u; z) fits in V is evaluated by DA, where z is random noise” teaches all of the translations are done randomly.
[Yi, page 6, Figure 6] “Experimental results for translating Chinese paintings to oil paintings (without GT available). The background grids shown in the GAN results imply that the outputs of GAN are not as stable as those of DualGAN”, teaches a randomly modified color of the first pattern,
[Yi, page 8, Figure 8] “Map→aerial photo translation. Without image correspondences for training, DualGAN may map the orange-colored interstate highways to building roofs with bright colors. Nevertheless, the DualGAN results are sharper than those from GAN and cGAN” and [Yi, page 8, Figure 9] “Results for aerial photo→map translation. DualGAN performs better than GAN, but not as good as cGAN. With additional pixel correspondence information, cGAN performs well in terms of labeling local roads, but still cannot detect interstate highways” teaches a randomly modified dashed line of the first data pattern to a continuous line, a randomly modified color of the first pattern, and a randomly removed line pattern from the first pattern. ).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava, Ceccaldi, Jiang, and Yi to use the randomly modified data sample comprises one or more of a randomly modified dashed line of the first data pattern to a continuous line, a randomly modified color of the first pattern, and a randomly removed line pattern from the first pattern of Yi to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to enable the system to handle a wider variety of types of images.
Regarding claim 10, Shrivastava teaches:
wherein the providing pairs of data samples ([Shrivastava, column 12, line 48-54] Given pairs of synthetic and real images, generator 120 may be configured to minimize the l.sub.1 or l.sub.2 norm of the image difference 460 between the original synthetic image and the refined synthetic image).
Shrivastava in view of Ceccaldi in view of Jiang does not specifically disclose:
wherein the providing pairs of data samples comprises: providing a set of images with patterns, determining at least one pattern to be modified, randomly modifying the at least one pattern of the images using a random number generator, and relating one out of the set of images and a related image with the at least one pattern defining one of the pairs comprising the base data sample and the modified data sample.
Yi teaches:
wherein the providing pairs of data samples comprises: providing a set of images with patterns ([Yi, page 3, Figure 1] “Figure 1: Network architecture and data flow chart of DualGAN for image-to-image translation”, shows the Generators and the Discriminator receiving a pair of images with patterns), determining at least one pattern to be modified ([Yi, page 5, left column, Figure 3] “Figure 3: Results of label!facade translation. DualGAN faithfully preserves the structures in the label images, even though some labels do not match well with the corresponding photos in finer details. In contrast, results from GAN and cGAN contain many artifacts. Over regions with labelphoto misalignment, cGAN often yields blurry output (e.g.,the roof in second row and the entrance in third row)”, shows that the DualGAN successfully determines which pattern to be preserved and which pattern to be modified), randomly modifying the at least one pattern of the images using a random number generator ([Yi, page 3, left column, line 5-18] “As shown in Fig. 1, image u 2 U is translated to domain V using GA. How well the translation GA(u; z) fits in V is evaluated by DA, where z is random noise, and so is z0 that appears below. GA(u; z) is then translated back to domain U using GB, which outputs GB(GA(u; z); z0) as the reconstructed version of u. Similarly, v 2 V is translated to U as GB(v; z0) and then reconstructed as GA(GB(v; z0); z). The discriminator DA is trained with v as positive samples and GA(u; z) as negative examples, whereas DB takes u as positive and GB(v; z0) as negative. Generators GA and GB are optimized to emulate “fake” outputs to blind the corresponding discriminators DA and DB, as well as to minimize the two reconstruction losses kGA(GB(v; z0); z)vk and kGB(GA(u; z); z0) uk”, the translation from an image to an image uses a random noise), and relating one out of the set of images and a related image with the at least one pattern defining one of the pairs comprising the base data sample and the modified data sample ([Yi, page 5, right column, 4th -5th paragraph] “In many cases, DualGAN also compares favorably over the supervised cGAN in terms of sharpness of the outputs and faithfulness to the input images; see Figures 2, 3, 4, 5, and 8. This is encouraging since the supervision in cGAN does utilize additional image and pixel correspondences. On the other hand, when translating between photos and semantic-based labels, such as map$aerial and label$facades, it is often impossible to infer the correspondences between pixel colors and labels based on targeted distribution alone. As a result, DualGAN may map pixels to wrong labels (see Figures 9 and 10) or labels to wrong colors/textures (see Figures 3 and 8). Figures 6 and 7 show image translation results obtained using the two unlabeled datasets, including oil<->Chinese, plastic->metal, metal->stone, leather->fabric, as well as wood<->plastic”, Yi teaches finding correspondence (relation) between labels and pixel colors. Figure 9 also shows the GT, DualGAN, GAN), and cGAN finds the map images corresponds to the input image).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava, Ceccaldi, Jiang, and Yi to use the method of randomly modifying a dataset using random number generator of Yi to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to make developing a random textual data generator easier as random number generator is one of the easiest way to introduce randomness to the system.
Claim 18 is a system claim having similar limitation to the method claim 10. Therefore, it is rejected with same rationale as claim 10 above.
Claim 5 and 15 are rejected under 35 U.S.C. 103 over Shrivastava (US 11475276 B1) in view of Ceccaldi (US 20190046068 A1), in view of Jiang (US 20200042613 A1), and further in view of Elgammal (US 20190392624 A1).
Regarding claim 5, Shrivastava in view of Ceccaldi, in view of Jiang teaches:
wherein the discriminator is a neural network having as many input nodes as the generator, has output nodes ([Ceccaldi, The first figure, 411, ‘Discriminator Network’] shows the number of nodes decreasing from the input to the output, [Ceccaldi, 0040] “Various units or layers may be used, such as convolutional, pooling (e.g., max pooling), deconvolutional, fully connected, or other types of layers. Within a unit or layer 435, any number of nodes is provided. For example, 100 nodes are provided. Later or subsequent units may have more, fewer, or the same number of nodes. In general, for convolution, subsequent units have more abstraction. For example, the first unit provides features from the image, such as one node or feature being a line found in the image. The next unit combines lines, so that one of the nodes is a corner. The next unit may combine features (e.g., the corner and length of lines) from a previous unit so that the node provides a shape indication. For transposed-convolution to reconstruct, the level of abstraction reverses. Each unit or layer 435 in the encoder 401 reduces the level of abstraction or compression while each unit or layer 435 in the decoder increases the level of abstraction”).
Shrivastava in view of Ceccaldi and further in view of Jiang does not specifically teach the discriminator having two output nodes.
Elgammal teaches:
the generator having having two output nodes ([Elgammal, 0043] “In order to “train” the system, all of the works of art, and each of their associated labels, included in source blocks 110, are passed through block 108 for presentation as digitized data to discriminator 100. Discriminator 100 has two output nodes 112 and 114”).
Before the effective filing date of the invention to a person of ordinary skill in the art, it would have been obvious, having the teachings of Shrivastava, Ceccaldi, Jiang, and Elgammal to use discriminator having two output nodes of Elgammal to implement the autonomous data modification system of Shrivastava. The suggestion and/or motivation for doing so is to compress the input feature so that it can output either positive or negative.
Claim 15 is a system claim having similar limitation to the method claim 4 and 5. Therefore, it is rejected with same rationale as claim 4 and 5 above.
Response to Arguments
Applicant's arguments filed 01/27/2025 have been fully considered but they are not persuasive.
Response to Arguments under 35 U.S.C. 101
[Remarks, page 9] Applicant asserts that the “training, without the use of labeled or annotated data, the generator for building a model of the generator using an adversarial training method” cannot practically be performed in the human mind.
Examiner’s response:
Examiner agrees that the “training, without the use of labeled or annotated data, the generator for building a model of the generator using an adversarial training method” cannot practically be performed in the human mind. However, the limitation is directed to mere instructions to apply an exception using a computer MPEP 2106.05(f), because the additional limitations provided only a result-oriented solution and lacked details as to how the computer performed the modifications, which was equivalent to the words "apply it". 850 F.3d at 1341-42; 121 USPQ2d at 1947-48 (citing Electric Power Group., 830 F.3d at 1356, 1356, USPQ2d at 1743-44 (cautioning against claims "so result focused, so functional, as to effectively cover any solution to an identified problem")).
[Remarks, page 10-11] Applicant asserts that the improvements to computerized machine learning are patent eligible under Section 101, as recognized by the PTAB in Ex parte Desjardins, Appeal 2024-000567, Decision on Request for Rehearing, BPAI September 26, 2025 (hereinafter ‘Desjardins’). Furthermore, applicant asserts that the claims include the components or steps of the invention that provide the improvement described in the spec [0031]-[0047] especially [0035]. [Remarks, page 11-13] In response to the interview of October 16, 2025, the applicant asserts that according to MPEP 2106.04(d)(1) and the present case, the operations disclosed in the claim improves the relevant existing technology of computerized machine learning using novel techniques. Applicant further asserts that the present specification explicitly recites “in order to boost the performance, improve the training convergence and decouple the architectures of the generator and discriminator, the Wassertein loss function is used instead of the adversarial” how the generator and the discriminator are decoupled during the cited operation and the training is performed “without the use of labeled or annotated data.” Additionally, the applicant asserts that the discriminator is a PatchGAN, thus the specification explicitly sets forth improvements with detail necessary to be apparent to a person of ordinary skill in the art.
Examiner’s response:
Examiner respectfully disagrees. The applicant is citing PTAB decisions that are not analogous to the current application. The specification 21 of Desjardins clearly disclose the components or steps that provide the improvement: “For example, by maintaining a single instance of a model rather than multiple different instances of a model each having different parameter values, only one set of parameters needs to be stored rather than multiple different parameter sets, reducing the amount of storage space required while maintaining acceptable performance on each task” and the claim clearly disclose the improvement in “computing, based on the first values of the plurality of parameters determined by training the machine learning model on the first machine learning task … training the machine learning model on the second machine learning task by training the machine learning model on the second training data”. However, the instant specification [0031]-[0047] set forth the improvement but in a conclusory manner. Spec [0032], [0034] and [0041] merely explains the conclusion that using Wassertein loss instead of the adversarial and using PatchGAN as the discriminator promise the best result with a fast convergence during training. What is the structure of PatchGAN? How does the Wassertein loss and PatchGAN promise the best result and fast convergence? The specification does not provide any details beyond the conclusion that the Wassertein loss and PatchGAN improves the architectures of the generator and the discriminator.
According to MPEP 2106.04(d)(1), the specification must describe the invention such that the improvement would be apparent to one of ordinary skill in the art. Conversely, if the specification explicitly sets forth an improvement but in a conclusory manner (i.e., a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art), the examiner should not determine the claim improves technology.
The claim must recite the improvement (achieving higher performance than traditional methods on hard speech recognition tasks) with the detail necessary to be apparent to a person of ordinary skill in the art. The paragraph [0032] of the specification describes that the proposed super resolution GAN may improve the training convergence and allowing a decoupling the architectures of the generator and the discriminator. The applicant asserted in [Remarks, page 11] that “using a trained neural network, along with a language model, achieved higher performance than traditional methods on hard speech recognition tasks.” The claim fails to recite how the generator and the discriminator are decoupled during the operation, and fails to recite how the language model is used along with a trained neural network, which are declared as improvements in paragraph [0032] and [Remarks, page 11]. The amended limitation “generating, using the trained generator, an additional text-based data sample” seems to recite reusing of the generator based on different dataset, and fails to recite how the language model and the trained generator are connected.
Additionally, MPEP 2106.05(a) states that a claim whose entire scope can be performed mentally, cannot be said to improve computer technology. Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 120 USPQ2d 1473 (Fed. Cir. 2016) (a method of translating a logic circuit into a hardware component description of a logic circuit was found to be ineligible because the method did not employ a computer and a skilled artisan could perform all the steps mentally). Similarly, a claimed process covering embodiments that can be performed on a computer, as well as embodiments that can be practiced verbally or with a telephone, cannot improve computer technology. See RecogniCorp, LLC v. Nintendo Co., 855 F.3d 1322, 1328, 122 USPQ2d 1377, 1381 (Fed. Cir. 2017) (process for encoding/decoding facial data using image codes assigned to particular facial features held ineligible because the process did not require a computer). Since the entire scope of the claim can be performed mentally and the additional elements of providing a pair of data and training the generator and discriminator is not sufficient to amount to significantly more than the judicial exception, the claim cannot be said to improve computer technology.
Accordingly, arguments to independent claim 1 is not persuasive and similarly the arguments to independent claims 13 and 20 are not persuasive.
Response to Arguments under 35 U.S.C. 103
[Remarks, page 16-17 and 19-20] In response to the Examiner’s assertion in the Office Action dated May 19, 2025, the applicant asserts that the ‘pose’ and /or ‘pixels’ of the images of Shrivastava correspond to a rendered image and the applicant finds no disclosure in Shrivastava that the ‘pose of the hands’ is a pattern of the data sample of Shrivastava that is discernible by the discriminator of Shrivastava, and thus, neither the “pose of hands” pattern nor the pixels of the images anticipate or render obvious the claimed modified data sample that has a first pattern of the plurality of patterns that is modified.
Examiner’s response:
Examiner respectfully disagrees. The instant specification [0044] “the modified data sample may comprise, in contrast to the relating data samples continuous lines instead of dashed lines, black- and-white instead of equivalent colored patterns, text-less patterns instead of patterns with text, and line-free images instead of mixed line/text images”, spec [0079] “With acoustic or sound data, a pattern to the modified could, e.g., be the language accent so that the proposed method can be used to modify the accent in a recording” and spec [0080] “With text data, a pattern to be modified could, e.g., be the use of particular expressions and sentences so that the proposed method can change it with other expressions keeping the semantic meaning in the sentence” clearly disclose that the pattern does not have to be a ‘repeated design’. If “the use of particular expressions and sentences so that the proposed method can change it with other expressions keeping the semantic meaning in the sentence” can be interpreted as the ‘pattern’, then the “particular image/pose that the proposed method can change it while keeping the pose” should be interpreted as the pattern as well. [Shrivastava, Fig. 2 and Fig. 4] clearly shows the image ’pattern’ and modifying the pattern from 220 to 210 using the learning method disclosed in [Shrivastava, col 11, line 57 – col 12, line 66].
Accordingly, arguments to independent claim 1 is not persuasive and similarly the arguments to independent claims 13 and 20 are not persuasive.
[Remarks, page 17] In response to the Examiner’s assertion in the Office Action dated May 19, 2025, the applicant asserts that the training of Ceccaldi is not disclosed to be applicable to the machine learning techniques of Shrivastava and the Examiner has provided no evidence to support such an assertion. Moreover, the applicant asserts that Shrivastava performs training using labeled data.
Examiner’s response:
Examiner respectfully disagrees. The training of Ceccaldi is applicable to the machine learning techniques of Shrivastava as both methods requires a generator neural network that generates synthetic data and a discriminator neural network that distinguishes synthetic data and original data. The summary in the spec [0005] also discloses a generator and a discriminator.
Even though the first reference Shrivastava discloses training process with labeled data, the combination of the first reference and the secondary reference Ceccaldi clearly disclose training without the use of labeled or annotated data in paragraph [Ceccaldi, 0045]. The paragraph 0045 discloses training using the loss function from both the decoder network and an adversarial network trained to discriminate between features of the encoder network 401. The training process does not require any labeled input data.
Accordingly, arguments to independent claim 1 is not persuasive and similarly the arguments to independent claims 13 and 20 are not persuasive.
[Remarks, page 18] In response to the Examiner’s assertion in the Office Action dated Oct 25, 2024, the applicant asserts that Shrivastava does not disclose or suggest training a generator for building a model of the generator using an adversarial training method that inputs both samples of the pairs of data samples into the generator thus combining Shrivastava with a reference (Jiang) does not cure the deficiency in Shrivastava.
Examiner respectfully disagrees. Examiner respectfully disagrees. Shrivastava clearly disclose training a generator for building a model of the generator using an adversarial training method that inputs both samples of the pairs of data sample into the generator. According to [Shrivastava, column 6, line 20 - 42], the generator which includes a synthesizer 110 and a generator 120, receives two input data comprising a real image and a synthetic image. The real image is interpreted as the base data, and the synthetic image and the refined synthetic data 430 are interpreted as the modified data. The generator generates refined synthetic image that goes into the discriminator based on the real image and the synthetic image pair which are further disclosed in [Shrivastava, col 12, line 48-54]. Synthesizer 110 which is a part of the generator generates synthetic data based on a set of training data including labeled real images. The generator 120 receives refined synthetic data 430 which is a modified data to generate a new synthetic data. The generator is trained based on a set of training data including real images received through the synthesizer and synthetic images, which are further disclosed in [Shrivastava, col 12, line 48-54]. Additionally, the paragraph discloses that the training process is performed by utilizing adversarial networks. In regards to the limitation ‘training, without the use of labeled or annotated data’, Ceccaldi clearly discloses training without the use of labeled or annotated data in paragraph [0045] as discussed above.
Accordingly, arguments to independent claim 1 is not persuasive and similarly the arguments to independent claims 13 and 20 are not persuasive.
35 U.S.C. 103 rejection - Claims 12 and 19
[Remarks, page 22-23] In response to the Examiner’s assertion in the Office Action dated May 19, 2025, the applicant asserts that not require a real image y_i as input data during the training process does not anticipate or render obvious the limitation: at least one of the base data samples lacks a corresponding modified data sample. Moreover, the applicant asserts that Shrivastava’s disclosure that synthetic images may not have a corresponding real image does not infer that at least one of the base data samples lacks a corresponding modified data sample, thus Shrivastava does not disclose or suggest “wherein the at least one of the base data samples lacks a corresponding modified data sample.”
Examiner’s response:
Examiner respectfully reiterates the arguments in the Office Action dated May 19, 2025. [Shrivastava, col 12, line 33-41; and Fig. 4] the real data 440 inputs to the discriminator to train both the generator and discriminator does not have any corresponding input synthetic image, because the refined synthetic data 430 corresponds to the synthetic data 420 and does not correspond to the real data 440. The real data 440 is simply used to refine the synthetic image to the corresponding refined synthetic image. Moreover, [Shrivastava, col 16, line 54 – col 17, line 5] indicates that the adversarial real data that is used in an adversarial net refinement process may not have to include actual examples of the objects (e.g., or other data types) that are trying to be learned, and [col 16, line 65 – col 17, line 5] many variations (modified) of that object which does not correspond to the real data may be used.
Accordingly, arguments to claims 12 and 19 are not persuasive.
35 U.S.C. 103 rejection - Claims 3, 6, 8-10 and 16-18
[Remarks, page 24-25] Applicant asserts that Yi does not disclose or suggest randomly modifying a color of the first pattern, wherein the random modification leaves at least one pattern of the plurality of patterns of the modified data sample unmodified.
Examiner’s response:
Examiner respectfully disagrees that translating an image u using GA(u;z), translating Chinese paintings to oil paintings, a modified image data that retains an original flying bird pattern, and an image that retains the original pattern of a bird sitting on a branch do not render obvious the four modifications of randomly modified dashed line to a continuous line, randomly modified color, randomly removed text pattern, and a randomly removed line pattern. The arguments do not sufficiently address why Yi fails to disclose the modifications disclosed in Claim 9.
First, [Yi, page 3, left col, line 5-7] “As shown in Fig. 1, image u 2 U is translated to domain V using GA. How well the translation GA(u; z) fits in V is evaluated by DA, where z is random noise” teaches all of the translations are done randomly.
Second, [Yi, page 6, Figure 6] “Experimental results for translating Chinese paintings to oil paintings (without GT available). The background grids shown in the GAN results imply that the outputs of GAN are not as stable as those of DualGAN”, teaches a randomly modified color of the first pattern,
[Yi, page 8, Figure 8] “Map→aerial photo translation. Without image correspondences for training, DualGAN may map the orange-colored interstate highways to building roofs with bright colors. Nevertheless, the DualGAN results are sharper than those from GAN and cGAN” and [Yi, page 8, Figure 9] “Results for aerial photo→map translation. DualGAN performs better than GAN, but not as good as cGAN. With additional pixel correspondence information, cGAN performs well in terms of labeling local roads, but still cannot detect interstate highways” teaches a randomly modified dashed line of the first data pattern to a continuous line, a randomly modified color of the first pattern, and a randomly removed line pattern from the first pattern.
Accordingly, arguments to claims 3, 6, 8-10 and 16-18 are not persuasive.
35 U.S.C. 103 rejection - Claims 5 and 15
According to the discussions above, arguments to claims 1 and 13 are not persuasive. Therefore, claims 5 and 15, which depend directly from claims 1 and 13, are not patentable.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US-20230177333-A1 (This prior art teaches generating modified (synthesized) data based on the base data to train a machine learning model)
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUN KWON whose telephone number is (571)272-2072. The examiner can normally be reached Monday – Friday 7:30AM – 4:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUN KWON/Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127
Prosecution Insights