DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 12/01/2025 has been entered. Claims 1-20 remain pending in the application.
Response to Arguments
Applicant’s arguments, filed 12/01/2025, with respect to the rejections of the claims under 103 have been fully considered and are persuasive because of the amendments. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Noguchi et al., "Image Generation from Small Datasets via Batch Statistics Adaptation", as provided by the applicant in the IDS in view of Dumoulin et al. “A Learned Representation for Artistic Style" in view of Kaist et al. “Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs” in view of Rahman Siddiquee et al. (US Pub. 2020/0364477) and further in view of Vineet (US Pub. 2021/0142107).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 8, 12, 15 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Noguchi et al., "Image Generation from Small Datasets via Batch Statistics Adaptation", as provided by the applicant in the IDS in view of Dumoulin et al. “A Learned Representation for Artistic Style" in view of Kaist et al. “Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs” in view of Rahman Siddiquee et al. (US Pub. 2020/0364477) and further in view of Vineet (US Pub. 2021/0142107).
Regarding claim 1, Noguchi teaches a method comprising:
accessing a pre-trained generative adversarial network (GAN) trained on a primary image domain [page 3, Col. 2, paragraph 4, "In this paper, we propose a method for adapting pre trained generative models to datasets of different domains"; page 5, Col. 2, paragraph 6, "we used the SNGA projection model pre-trained on the ImageNet 1K dataset"];
generating a fine-tuned GAN based on the pre-trained GAN [page 2, Col. 1, paragraph 3, "we propose a novel method for transferring a pre-trained generative model to realize image generation from a small dataset"; page 3, Col. 2, paragraph 5, "To use the prior know ledge obtained, the adapted generator is trained from the weights acquired in the pre-trained generator"], by performing operations comprising:
generating layers for the fine-tuned GAN [page 5, Col. 2, paragraph 4, "This method ... updates all parameters or only a few layers of the generator]; and
adjusting weights of neural network layers of the pre-trained GAN using weights of neural network layers of the fine-tuned GAN [page 8, Col. 2, paragraph 2, "by changing the scale and shift parameters and latent vector, we can perform smooth morphing between different domains", As part of their method for generating a finetuned GAN Noguchi teaches updating scale and shift parameters within the generator to adjust the output of each layer in order to change which filters in the convolution layer affect the image and to what degree"]; and
storing, by one or more processors, the fine-tuned GAN [Figure 4 demonstrates training of model, as well as its use, thus it is inherent that the fine-tuned GAN was stored by one or more processors at some point].
Noguchi does not explicitly teach
identifying input data of the fine-tuned GAN, the input data comprising a set of manipulation conditions and a set of images from a secondary image domain, the secondary image domain being different from the primary image domain;
generating a set of style vectors based on the set of manipulation conditions;
generating a set of transformed style vectors based on the set of style vectors;
identifying first transformation blocks of a generator network in the pre-trained GAN that are affected by the set of transformed style vectors;
in response to identifying the first transformation blocks, freezing the first transformation blocks of the generator network in the pre-trained GAN affected by the set of transformed style vectors;
training second transformation blocks of the pre-trained GAN affected by the set of style vectors;
generating additional layers for the fine-tuned GAN; and
training the additional layers of the fine-tuned GAN based on the input data and the set of style vectors;
Dumoulin teaches
identifying input data of the fine-tuned GAN, the input data comprising a set of manipulation conditions and a set of images from a secondary image domain, the secondary image domain being different from the primary image domain [Fig. 2; page 4, paragraph 4, "The conditional network is given both a content image and the identity of the style to apply and produces a pastiche corresponding to that style];
generating a set of style vectors based on the set of manipulation conditions [page 4, paragraph 4, "The conditional network is given both a content image and the identity of the style to apply and produces a pastiche corresponding to that style ... we found a very surprising fact about the role of normalization in style transfer networks: to model a style, it is sufficient to specialize scaling and shifting parameters after normalization to each specific style. In other words, all convolutional weights of a style transfer network can be shared across many styles, and it is sufficient to tune parameters for an affine transformation after normalization for each style. We call this approach conditional instance normalization"];
training the layers of the fine-tuned GAN based on the input data and the set of style vectors [page 4, paragraph 5,"The goal of the procedure is transform a layer's activations x into a normalized activation z specific to painting style s. Building off the instance normalization technique proposed in Ulyanov et al. (2016b ), we augment the γ and β parameters so that they 're N x C matrices, where N is the number of styles being modeled and C is the number of output feature maps. Conditioning on a style is achieved as follows: z =
PNG
media_image1.png
38
130
media_image1.png
Greyscale
where μ and ơ are x's mean and standard deviation taken across spatial axes and γs and βs are obtained by selecting the row corresponding to s in the γ and β matrices (Figure 3)"; Figure 2; page 2, paragraph 1, “Figure 1: Pastiches produced by a style transfer network trained on 32 styles chosen for their variety"];
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include a set of manipulation conditions and a set of images from a secondary image domain in input data, the secondary image domain being different from the primary image domain, as well as by generating a set of style vectors based on the set of manipulation conditions and training the identified layers of the fine-tuned GAN based on the input data and the set of style vectors of Dumoulin. Doing so would allow the network the learn multiple artistic styles, as well as allow for the combination of multiple styles for the purpose of image generation/modification (Dumoulin, page 2, paragraphs 2-3).
Noguchi and Dumoulin do not explicitly teach
generating a set of transformed style vectors based on the set of style vectors;
identifying first transformation blocks of a generator network in the pre-trained GAN that are affected by the set of transformed style vectors;
in response to identifying the first transformation blocks, freezing the first transformation blocks of the generator network in the pre-trained GAN affected by the set of transformed style vectors;
training second transformation blocks of the pre-trained GAN affected by the set of style vectors;
generating additional layers for the fine-tuned GAN (emphasis added); and
training the additional layers of the fine-tuned GAN (emphasis added);
Kaist teaches
generating a set of transformed style vectors based on the set of style vectors [page 2, Col. 1, paragraph 1, “we fine-tune the StyleGAN architecture, which is pre-trained on FFHQ, onto Animal Face and Anime Face dataset”];
freezing the first transformation blocks of the pre-trained GAN affected by the set of transformed style vectors [page 2, Col. 1, paragraph 1, “We demonstrate the effectiveness of the simple baseline, dubbed FreezeD … we fine-tune the StyleGAN architecture, which is pre-trained on FFHQ, onto Animal Face and Anime Face dataset”; page 2, Col. 2, paragraph 3, “FreezeD (our proposed baseline): We find that simply freezing the lower layers of the discriminator and only fine-tune the upper layers performs surprisingly well. We call this simple yet effective baseline as FreezeD”];
training second transformation blocks of the pre-trained GAN affected by the set of style vectors [page 2, Col. 2, paragraph 3, “FreezeD (our proposed baseline): We find that simply freezing the lower layers of the discriminator and only fine-tune the upper layers performs surprisingly well. We call this simple yet effective baseline as FreezeD”; page 4, section 3.2. Conditional GAN and section 4. Conclusion, “We use the SNGAN-projection architecture pre-trained on ImageNet dataset, and fine-tune it on Oxford Flower, CUB-200-2011, and Caltech-256 datasets … Figure 3 visualizes the samples generated using the model trained by fine-tuning and FreezeD … We freeze the discriminator until f3, 2, 1g layers for (Oxford Flower, CUB-200-2011, Caltech-256 datasets) … We have introduced a simple yet effective baseline, FreezeD, for transfer learning of GANs. FreezeD splits the discriminator into a feature extractor and a classifier and then fine-tune the classifier only”; examiner interprets the upper layers or the classifier of the discriminator as second transformation block that is trained/fine-tuned when the other block is frozen, and interprets the layers of the GAN as the transformation blocks; page 1, Introduction, Col. 2, last paragraph, “the lower layers of the discriminator learn generic features of images while the upper layers learn to classify whether the image is real or fake based on the extracted features”];
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include generating a set of transformed style vectors based on the set of style vectors, freezing first transformation blocks of the pre-trained GAN affected by the set of transformed style vectors, and training second transformation blocks of the pre-trained GAN affected by the set of style vectors of Kaist. Doing so would help improving both the performance and the stability of the model (Kaist, page 3, Col. 2, paragraph 1).
Noguchi, Dumoulin and Kaist do not explicitly teach
identifying first transformation blocks of a generator network in the pre-trained GAN that are affected by the set of transformed style vectors;
in response to identifying the first transformation blocks, freezing the first transformation blocks of the generator network;
generating additional layers for the fine-tuned GAN (emphasis added); and
training the additional layers of the fine-tuned GAN (emphasis added);
Rahman Siddiquee teaches
identifying first transformation blocks of a generator network in the pre-trained GAN that are affected by the set of transformed style vectors [Fig. 1A, paragraph 0026, “referring to process 110, the generator network can take as inputs a real image x and a domain of the target fake image, cy- For example, in an instance in which the images correspond to images of faces of people, real image x can be an image of a face of a person who has black hair and is male, and cy can correspond to a domain of the generated fake image y', where y' is associated with at least one domain that is different than the domains associated with real image x. As a more particular example, in an instance in which real image x is associated with the domains of black hair and male, cy can be blonde hair”; paragraph 0017, “a generator network can be trained to generate a fake image that is a translation of an input real image with respect to a particular domain … in an instance in which images correspond to faces of people, a generator network can be trained to take, as an input, an image of a face of a person who has black hair and generate a fake image that corresponds to an image of the person with blond hair … in an instance in which images correspond to medical images of a particular portion of a body, a generator network can be trained to take, as an input, a real image of the portion of the body and generate a translated image corresponding to a healthy domain … the mechanisms can further generate a difference map that indicates a difference between the generated, translated image (e.g., the translated image in the healthy domain) and the input image”; It can be seen that the generator comprises transformation blocks/ mechanisms which generate the difference between the generated/translates image and the input image, and those transformation blocks/ mechanisms are affected by the set of inputs];
in response to identifying the first transformation blocks, freezing the first transformation blocks of the generator network [Fig. 1A, paragraph 0026, “as shown by the solid box around the generator network G, in process 110, the generator network can be held constant or frozen while the discriminator network is being trained”];
Rahman Siddiquee further teaches
training second transformation blocks of the pre-trained GAN [paragraph 0026, “the generator network can be held constant or frozen while the discriminator network is being trained”];
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include identifying first transformation blocks of a generator network in the pre-trained GAN that are affected by the set of transformed style vectors, and in response to identifying the first transformation blocks, freezing the first transformation blocks of the generator network of Rahman Siddiquee. Doing so would help training a Generative Adversarial Network to generate images and generating a new image by translating an input image (Rahman Siddiquee, 0015).
Noguchi, Dumoulin, Kaist and Rahman Siddiquee do not explicitly teach
generating additional layers for the fine-tuned GAN (emphasis added); and
training the additional layers of the fine-tuned GAN (emphasis added);
Vineet teaches
generating additional layers for the fine-tuned GAN [paragraph 0030, “datasets. One approach to training such high capacity models involves two stages: a pre-training stage where the CNN model is trained on a simpler image semantic classification task (i.e. the model learns to classify images as a whole based on their content), and a fine-tuning (transfer learning) stage in which the pre-trained model is fine-tuned for other tasks such as semantic segmentation”; paragraph 0063, “Transfer learning/fine-tuning is known per se. As noted above, state of the art segmentation networks have been trained by fine-tuning a CNN that has been pre-trained on an image classification task … A typical approach is to replace one or more of the final layers of the CNN with new up sampling layer(s), which in tum are trained on a relatively small amount of segmentation data”; paragraph 0072, “Following pre-training, the architecture of the CNN is modified by discarding the output layer 410 is discarded and replacing it with a new output layer”]; and
training the additional layers of the fine-tuned GAN [paragraph 0063, “Transfer learning/fine-tuning is known per se. As noted above, state of the art segmentation networks have been trained by fine-tuning a CNN that has been pre-trained on an image classification task … A typical approach is to replace one or more of the final layers of the CNN with new up sampling layer(s), which in tum are trained on a relatively small amount of segmentation data”; paragraph 0072, “Following pre-training, the architecture of the CNN is modified by discarding the output layer 410 is discarded and replacing it with a new output layer … The weights of the new output layer are then trained to match the outputs of the new output layer 412 to the ground truth segmentation labels assigned to the finetuning data, whilst freezing the weights of the original layers 402-408”];
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include generating additional layers for the fine-tuned GAN and training the additional layers of the fine-tuned GAN of Vineet. Doing so would help accurately performing a desired task, and learning the more complex image processing task using transfer learning (Vineet, 0032).
Regarding claim 3, Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet teach the method of claim 1.
Noguchi teaches
identifying existing layers of the fine-tuned GAN for training [page 4, Col.1, paragraph 1, "We introduce the scale and shift parameters for each channel of the hidden layer distribution of each layer ... and update only these parameters to conduct an adaptation"]; and
Dumoulin teaches
training the existing layers of the fine-tuned GAN based on the input data and the set of style vectors [page 4, paragraph 5,"The goal of the procedure is transform a layer's activations x into a normalized activation z specific to painting style s. Building off the instance normalization technique proposed in Ulyanov et al. (2016b ), we augment the γ and β parameters so that they 're N x C matrices, where N is the number of styles being modeled and C is the number of output feature maps. Conditioning on a style is achieved as follows: z =
PNG
media_image1.png
38
130
media_image1.png
Greyscale
where μ and ơ are x's mean and standard deviation taken across spatial axes and γs and βs are obtained by selecting the row corresponding to s in the γ and β matrices (Figure 3)"; Figure 2; page 2, paragraph 1, “Figure 1: Pastiches produced by a style transfer network trained on 32 styles chosen for their variety”];
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include training the identified layers of the fine-tuned GAN based on the input data and the set of style vectors of Dumoulin. Doing so would allow the network the learn multiple artistic styles, as well as allow for the combination of multiple styles for the purpose of image generation/modification (Dumoulin, page 2, paragraphs 2-3).
Regarding claim 5, Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet teach the method of claim 1.
Kaist teaches
generating the fine-tuned GAN further comprises: excluding at least one layer of the fine-tuned GAN from training [page 2, Col. 1, paragraph 1, “We demonstrate the effectiveness of the simple baseline, dubbed FreezeD … we fine-tune the StyleGAN architecture, which is pre-trained on FFHQ, onto Animal Face and Anime Face dataset”; page 2, Col. 2, paragraph 3, “FreezeD (our proposed baseline): We find that simply freezing the lower layers of the discriminator and only fine-tune the upper layers performs surprisingly well. We call this simple yet effective baseline as FreezeD”].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include excluding at least one layer of the fine-tuned GAN from training of Kaist. Doing so would help improving both the performance and the stability of the model (Kaist, page 3, Col. 2, paragraph 1).
Regarding claims 8 and 12, claims 8 and 12 claim a system comprising a processor and a memory storing instructions that, when executed by the processor, configure the system to perform the method of claims 1 and 5, respectively. Noguchi et al. demonstrate possession of the code for the method claimed in claims 1 and 5 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation") and, through the presentation of the results of their experiments, the execution of this code by a processor at some point (Noguchi et al.: Figure 4). Claims 8 and 12 are thus rejected for reasons set forth in the rejections of claims 1 and 5, respectively.
Regarding claims 15 and 19, claims 15 and 19 claim a non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform the method of claims 1 and 5. Noguchi et al. demonstrate possession of the code for the method recited in claims 1 and 5 (Noguchi et al.: Abstract, "Code is available at github.com/nogu-atsu/small-dataset-image-generation"), the execution of this code by a processor at some point, and its existence on a non-transitory computer-readable storage
medium at some point through the presentation of the results of their experiments (Noguchi et al.: Figure 4). Claims 15 and 19 are thus rejected for reasons set forth in the rejections of claims 1 and 5, respectively.
Claims 2, 9-10 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Noguchi et al., in view of Dumoulin et al. in view of Kaist et al. in view of Rahman Siddiquee et al. in view of Vineet and further in view of Oxholm (US Pub. 2018/0357800 Al).
Regarding claim 2, Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet teach the method of claim 1.
Noguchi further teaches
accessing an image comprising a face and accessing a generator neural network of the fine-tuned GAN, the generator neural network trained to generate a modified image based on the image and condition data for the image [page 7, Col. 2, paragraph 1, “To investigate, we compared the transferred results to the anime face dataset and flower dataset adapted from a randomly initialized generator, a generator pre-trained on the face dataset [17], the LSUN bedroom dataset [36], and the ImageNet 1K. It can be inferred that a generator trained using the face and bedroom datasets has useful filters for image generation compared to a randomly initialized generator, although the diversity of the acquired filters is small compared to the generator trained using ImageNet”; Figure 8; Figure 14; Page 8, Col. 2, Paragraph 2, "From this, by changing the scale and shift parameters and latent vector, we can perform smooth morphing between different domains, similar to what is performed in [20, 4]. We conducted this domain morphing between "cheeseburger" in ImageNet generated by the pre-trained generator and each target dataset"].
Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet does not explicitly teach
accessing an image comprising a face from a computing device.
Oxholm teaches
accessing an image comprising a face from a computing device [paragraph 0021, “The input image may be transferred from a digital camera, downloaded from a website, received from a client device, etc. The style exemplar, which may be a photograph or a drawing, includes a set of varying-resolution versions of the same style exemplar (e.g., a 256x256 version, 512x512 version, and a 1024x1024 version”].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include accessing the image comprising a face from a computer device of Oxholm. Doing so would allow users such as creative professionals or content designers, through their devices, to utilize creative tools that leverage methods such as those of Noguchi et al. for image manipulation purposes (Oxholm: Paragraph [0034]).
Regarding claim 9, claim 9 recites a system comprising a processor and a memory storing instructions that, when executed by the processor, configure the system to perform the method of claim 2. Noguchi et al. demonstrate possession of the code for the method claimed in claim 2 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation") and, through the presentation of the results of their experiments, the execution of this code by a processor at some point (Noguchi et al.: Figure 4). Claim 9 is thus rejected for reasons set forth in the rejection of claim 2.
Regarding claim 16, claim 16 recites a non-transitory computer-readable storage medium, the computer readable storage medium including instructions that when executed by a computer, cause the computer to perform the method of claim 2. Noguchi et al. demonstrate possession of the code for the method recited in claim 2 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation"), the execution of this code by a processor at some point, and its existence on a non-transitory computer-readable storage medium at some point through the presentation of the results of their experiments (Noguchi et al.: Figure 4). Claim 16 is thus rejected for reasons set forth in the rejection of claim 2.
Regarding claim 10, Noguchi, Dumoulin, Kaist, Rahman Siddiquee, Vineet and Oxholm teach the system of claim 9.
Oxholm further teaches
causing presentation of the modified image on a graphical user interface of the computing device [paragraph [0064], "In some embodiments, these output operations include configuring a display device to render a graphical interface that includes the stylized output image and causing a display device to display the graphical interface with the stylized output image"].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include causing presentation of the modified image on a graphical user interface of the computing device of Oxholm. Doing so would enable those training or utilizing the GAN to view the results of the model (Oxholm: paragraph 0034).
Regarding claim 17, claim 17 recites a non-transitory computer-readable storage medium, the computer readable storage medium including instructions that when executed by a computer, cause the computer to perform the method of claim 10. Noguchi et al. demonstrate possession of the code for the method recited in claim 10 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation"), the execution of this code by a processor at some point, and its existence on a non-transitory computer-readable storage medium at some point through the presentation of the results of their experiments (Noguchi et al.: Figure 4). Claim 17 is thus rejected for reasons set forth in the rejection of claim 10.
Claims 4, 11 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Noguchi et al., in view of Dumoulin et al. in view of Kaist et al. in view of Rahman Siddiquee et al. in view of Vineet and further in view of He et al. (US Pub. 2021/0190882).
Regarding claim 4, Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet teach the method of claim 1.
Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet do not teach
decomposing changes learned by the fine-tuned GAN based on the set of transformed style vectors.
He teaches
decomposing changes learned by the fine-tuned GAN based on the set of transformed style vectors [paragraph 0051, “In the multi-stage transferring learning, a complete feature set satisfying the data to be diagnosed is formed through extraction of features of each step of the data sets, and all features are integrated based on a logic order from the bottom layer to the top layer through network training step by step. Through the multi-stage transferring learning, the features of the data to be diagnosed may be decomposed into feature modules.” Since Noguchi (as modified) teaches a process of fine-tuning of GANs to learn the optimal parameters based on the set of transformed style vectors (Kaist, page 1, Col. 2 and page 2, Col. 1, paragraph 1), and He teaches the parameters learned during the transferring learning is decomposed, therefore, the combination of Noguchi (as modified) and He teaches the above claim limitation].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include decomposing changes learned by the fine-tuned GAN based on the set of transformed style vectors of He. Doing so would help solving the problems that lacking of data sets to be diagnosed which include all features (He 0051).
Regarding claim 11, claim 11 recites a system comprising a processor and a memory storing instructions that, when executed by the processor, configure the system to perform the method of claim 4. Noguchi et al. demonstrate possession of the code for the method claimed in claim 4 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation") and, through the presentation of the results of their experiments, the execution of this code by a processor at some point (Noguchi et al.: Figure 4). Claim 11 is thus rejected for reasons set forth in the rejection of claim 4.
Regarding claim 18, claim 18 recites a non-transitory computer-readable storage medium, the computer readable storage medium including instructions that when executed by a computer, cause the computer to perform the method of claim 4. Noguchi et al. demonstrate possession of the code for the method recited in claim 4 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation"), the execution of this code by a processor at some point, and its existence on a non-transitory computer-readable storage medium at some point through the presentation of the results of their experiments (Noguchi et al.: Figure 4). Claim 18 is thus rejected for reasons set forth in the rejection of claim 4.
Claims 6, 13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Noguchi et al., in view of Dumoulin et al. in view of Kaist et al. in view of Rahman Siddiquee et al. in view of Vineet and further in view of Wang et al., "Transferring GANs: Generating Images from Limited Data" (published: 2018).
Regarding claim 6, Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet teach the method of claim 1.
Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet do not explicitly teach
wherein the pre-trained GAN comprises an image generator neural network and an image discriminator neural network, wherein the generating the fine-tuned GAN further comprises:
accessing an image associated with the set of manipulation conditions; and
updating the image discriminator neural network using the image and second residual data, wherein the second residual data is based on weights of a last layer of the image discriminator neural network and the set of manipulation conditions.
Wang teaches
wherein the pre-trained GAN comprises an image generator neural network and an image discriminator neural network [page 5, "To study the effect of domain transfer for GAN s we will use the WGAN-GP [24] architecture which uses ResNet in both generator and discriminator ... In a first experiment we consider the four possible combinations using a GAN pre-trained with Image Net and 1 00K samples of LSUN bedrooms as target dataset"], wherein the generating the fine-tuned GAN further comprises:
accessing an image associated with the set of manipulation conditions [page 11, Paragraph 1, "cGAN s allow us to condition the generative model on particular information such as classes, attributes, or even other images"]; and
updating the image discriminator neural network using the image and second residual data, wherein the second residual data is based on weights of a last layer of the image discriminator neural network and the set of manipulation conditions [page 11, Paragraph 2, "In this formulation, the discriminator has an 'auxiliary classifier' that outputs a probability distribution over classes P(C = y|x) conditioned on the input x. The objective function is then composed of the conditional version of the GAN loss LGAN (eq. (2)) and the log-likelihood of the correct class"].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include acquiring a pre trained image generator neural network and image discriminator neural network, accessing an image associated with the set of manipulation conditions and updating the image discriminator neural network using the image and second residual data, wherein said second residual data is based on weights of a last layer of the image discriminator neural network and the set of manipulation conditions of Wang. Doing so would allow for the training of models which result in lower Frechet Inception Distance (FID) scores and more recognizable images without having to rely on a larger training dataset (Wang et al.: Figure 1).
Regarding claim 13, Claim 13 recites a system comprising a processor and a memory storing instructions that, when executed by the processor, configure the system to perform the method of claim 6. Noguchi et al. demonstrate possession of the code for the method claimed in claim 6 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation") and, through the presentation of the results of their experiments, the execution of this code by a processor at some point (Noguchi et al.: Figure 4). Claim 13 is thus rejected for reasons set forth in the rejection of claim 6.
Regarding claim 20, Claim 20 recites a non-transitory computer-readable storage medium, the computer readable storage medium including instructions that when executed by a computer, cause the computer to perform the method of claim 6. Noguchi et al. demonstrate possession of the code for the method recited in claim 6 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation"), the execution of this code by a processor at some point, and its existence on a non-transitory computer-readable storage medium at some point through the presentation of the results of their experiments (Noguchi et al.: Figure 4). Claim 20 is thus rejected for reasons set forth in the rejection of claim 6.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Noguchi et al., in view of Dumoulin et al. in view of Kaist et al. in view of Rahman Siddiquee et al. in view of Vineet and further in view of Denli (US Pub. 2020/0183047).
Regarding claim 7, Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet teach the method of claim 1.
Noguchi, Dumoulin, Kaist, Rahman Siddiquee and Vineet do not explicitly teach
training the additional layers and residual data on the secondary image domain.
Denli teaches
training the additional layers and residual data on the secondary image domain [paragraph 0075, "In particular, responsive to acquiring additional data, the system may continue training (or re-training) the existing generative network or expand the existing generative network (e.g., increasing number of filters in a layer or adding new layers) in order to incorporate the additional data … an existing and previously-trained generative network may be expanded with additional layers and its expanded part may be only trained with the additional data while the previously-trained part of the generative network is fixed (e.g., not trained)"].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for generating image of Noguchi to include training the additional layers and residual data on the secondary image domain of Denli. Doing so would allow for previous learnings to be incorporated into a new expanded model as new data is incorporated and pre-trained features are tuned to a second domain (Denli, paragraph 0075, "This may also be referred as a transfer learning where the previous-learnings are transfer to the new expanded model while new data is incorporated in the generative network").
Regarding claim 14, Claim 14 recites a system comprising a processor and a memory storing instructions that, when executed by the processor, configure the system to perform the method of claim 7. Noguchi et al. demonstrate possession of the code for the method claimed in claim 7 (Noguchi et al.: Abstract, "Code is available at github.com/noguatsu/small-dataset-image-generation") and, through the presentation of the results of their experiments, the execution of this code by a processor at some point (Noguchi et al.: Figure 4). Claim 14 is thus rejected for reasons set forth in the rejection of claim 7.
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Vogels et al. (US Pub. 2018/0293713) describes a method for training the generator and the discriminator alternatingly, in one iteration, the generator may be frozen and the discriminator may be trained against it, and in a next iteration, the discriminator may be frozen, and the generator may be trained against the slightly better discriminator.
Isikdogan et al. (US Pub. 2020/0293870) describes a method and system for processing data and more particularly to partially-frozen neural networks for efficient computer vision systems.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRI T NGUYEN whose telephone number is 571-272-0103. The examiner can normally be reached M-F, 8 AM-5 PM, (CT).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, OMAR FERNANDEZ can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRI T NGUYEN/Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights