Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner notes the entry of the following papers:
Amended claims filed 5/21/2026.
Applicant’s arguments/remarks made in amendment filed 5/21/2026.
Claims 1-3, 5, 7-9, 11-16, 18, and 22-25 are amended. Claims 6, 10, and 19-21 are cancelled. Claims 26-28 are new. Claims 1-5, 7-9, 11-18, and 22-28 are presented for examination.
Response to Arguments
Applicant presents arguments. Each is addressed.
Applicant’s arguments that the prior art does not teach the amended claims are moot in view of new areas of the prior art found that teach the amended claims.
Applicant argues that “claims 14 and 24 are allowable for reasons including some of those discussed above in connection with claim 1.” (Remarks, page 9, paragraph 4, line 1.) However, claim 1 remains rejected. Independent claims 14 and 24 which recite similar limitations are similarly rejected. The dependent claims remain rejected at least for depending from rejected base claims.
Claim Objections
Claims 15-16 and 23 are objected to because of the following informalities: the claims recite the phrase “the the”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 7-9, 11, 13-15, 17-18, and 24-28 are rejected under 35 U.S.C. § 103 as being unpatentable over Liu, et al (Coupled Generative Adversarial Networks, herein Liu), and Larsen, et al (Autoencoding beyond pixels using a learned similarity metric, herein Larsen)
Regarding claim 1,
Liu teaches a computer-implemented method (Liu, Figure 1, and page 1, paragraph 1, line 1 “We propose coupled generative adversarial network (CoGAN) for learning a joint distribution of multi-domain images… It can learn a joint distribution with just samples drawn from the marginal distributions.” And, page 1, paragraph 2, line 1 “The paper concerns the problem of learning a joint distribution of multi-domain images from data.” And paragraph, 4, line 1 “CoGAN consists of a tuple of GANs, each for one image domain….We show that by enforcing a weight-sharing constraint the CoGAN can learn a joint distribution without existence of corresponding images in different domains.”
PNG
media_image1.png
441
1021
media_image1.png
Greyscale
In other words, CoGAN is a computer implemented method.), comprising:
Liu teaches a first domain and a second domain. (Liu, page 1, paragraph 4, line 1 “CoGAN consists of a tuple of GANS, each for one image domain.” And, page 1, paragraph 4, line 8 “CoGAN is for multi-image domains but, for ease of presentation, we focused on the case of two image domains in the paper. However, the discussions and analyses can be easily generalized to multiple image domains.” In other words, two image domains are a first domain and a second domain.)
[using a first encoder neural network to encode one or more first images of a first domain into a first latent code of a shared latent space; using a second encoder neural network to encode one or more second images of a second domain into a second latent code of the shared latent space], wherein
the one or more weights of the second encoder neural network are constrained using an equality constraint (Liu, page 1, paragraph 4, line 5 “By enforcing the layers that decode high-level semantics in the GANs to share the weights, it forces the GANs to decode the high-level semantics in the same way. The layers that decode low-level details then map the shared representation to image in individual domains for confusing the respective discriminative models.” Examiner notes, there is no mention of “constrained using an equality constraint” in the specification. The specification recites “the shared-latent space assumption is implemented using a weight sharing constraint, where the connection weights of one or more of the last layers in the encoder neural network 115 and the encoder neural network 105 are shared.” (Specification, paragraph [0035], line 2.) Further, claims 26-28 recite “…the equality constraint requires one or more final layer weights of the first encoder neural network to be equal to one or more final layer weights of the second encoder neural network.” Examiner further notes that if the weights are “shared” then for each of the respective networks that share the weights, they will be equal. Therefore, examiner is interpreting that “constrained using an equality constraint” means the one or more weights are shared. In other words, weight sharing constraint is the one or more weights of the second encoder neural network are constrained using an equality constraint.); and wherein
the first encoder neural network is configured to generate the first latent code in the shared latent space using the one or more constrained layer weights shared by the second encoder neural network (Liu, Figure 1, In other words, generator for GAN1 is first encoder neural network, generator for GAN2 is second encoder neural network, z is the shared latent space, g1(z) is first latent code, and weight sharing is one or more constrained layer weights shared by the second encoder neural network.)
[using a generator neural network to generate one or more translated images depicting one or more features of the one or more first images represented in the second domain based, at least in part, on the first latent code and the second latent code.]
Thus far, Liu does not explicitly teach using a first encoder neural network to encode one or more first images of a first domain into a first latent code of a shared latent space, using a second encoder neural network to encode one or more second images of a second domain into a second latent code of the shared latent space.
Larsen teaches using a first encoder neural network to encode one or more first images of a first domain into a first latent code of a shared latent space, producing a first latent code of the shared latent space (Larsen, page 2, column 1, paragraph 1, line 1“We combine VAEs and GANs into an unsupervised generative model that simultaneously learns to encode, generate and compare dataset samples.” And, page 2, column 1, paragraph 3, line 1 “A VAE consists of two networks that encode a data sample x to a latent representation z and decode the latent representation back to data space, respectively:
PNG
media_image2.png
28
525
media_image2.png
Greyscale
” In other words, encode is encoding, two networks (the first encodes) is first neural network in a first domain, encode a data sample x to a latent representation z is convert the first image to a shared latent space, and latent representation z is a first latent code.);
Larsen teaches using a second encoder neural network to encode one or more images of a second domain into a second latent code of the shared latent space (Larsen, page 2, column 1, paragraph 1, line 1“We combine VAEs and GANs into an unsupervised generative model that simultaneously learns to encode, generate and compare dataset samples.” And, page 2, column 1, paragraph 3, line 1 “A VAE consists of two networks that encode a data sample x to a latent representation z and decode the latent representation back to data space, respectively:
PNG
media_image2.png
28
525
media_image2.png
Greyscale
” In other words, encode is encoding, two networks (the second encodes) is second neural network in a second domain, encode a data sample x to a latent representation z is convert the second image to a shared latent space, and latent representation z is a second latent code. Doing the same steps again, is encoding by a second network, a second image to a shared latent space, producing a second latent code z.)
Larsen teaches using a generator neural network to generate one or more translated images depicting one or more features of the one or more first images represented in the second domain based, at least in part, on the first latent code and the second latent code. (Larsen, page 2, column 1, paragraph 5, line 1 “A VAE consists of two networks that encode a data sample x to a latent representation z and decode the latent representation back to data space respectively:
PNG
media_image3.png
29
579
media_image3.png
Greyscale
” In other words, decoder is third neural network, decode is generating wherein the first translated image is correlated with the first image and weight values of the third neural network are computed based on the first latent code.)
In order to motivate the combination of Liu and Larsen, a brief description of the underlying technology is necessary. A GAN (generative adversarial network) is a machine learning model in which two neural networks, called the generator and discriminator, respectively, compete with each other to become more accurate in their predictions. The generator, learns to generate plausible data. The generated instances become negative training examples for the discriminator. The discriminator learns to distinguish the generator’s fake data from real data. For example, in the case of images, this means discriminating between an image generated by the generator and an actual photograph. The discriminator penalizes the generator for producing implausible results.
Liu combines two GANS into a coupled generative adversarial network (CoGAN) to learn a joint distribution, between two or more domains. Liu does this to allow for unsupervised learning of a joint domain by enforcing the layers that decode high-level semantics in the GANs to share weights, thereby forcing the GANs to decode the high-level semantics in the same way. (See, Figure 1 of Liu.)
An autoencoder is a neural network that combines an encoder and a decoder. The encoder maps the input into a latent code and the decoder takes the latent code and produces an output that maps back to the input. The goal of an autoencoder is to learn a lower-dimensional representation (encoding) for a higher-dimensional data, typically for dimensionality reduction, by training the network to capture the most important parts of the input. The loss function used to train an autoencoder is called reconstruction loss, as it is a check of how well the image has been reconstructed from the input.
A Variational Autoencoder (VAE) performs the same function as the autoencoder, but instead of the encoder’s output being a latent vector, the encoder outputs the mean and the standard deviation for each latent variable. It does this in order to normalize the output and remove outliers. The loss function is the reconstruction loss, as with a typical autoencoder, combined with a similarity loss.
PNG
media_image4.png
353
682
media_image4.png
Greyscale
Larsen teaches a combination of VAE and GAN called a VAE/GAN. The VAE/GAN encodes the input with the encoder into the latent space, and then decodes the latent space to the reconstructed input. The reconstructed input is then used as the generator in the GAN where the reconstructed input is compared to a real image by the discriminator. In other words, when the VAE encoder-decoder is combined with the GAN generator-discriminator, the decoder of the VAE becomes the generator of the GAN. The loss is then promulgated back to the encoder. “The end result will be a method that combines the advantage of GAN as a high quality generative model and VAE as a method that produces an encoder of data into the latent space z.” (Larsen, page 2, column 2, paragraph 3, line 5.)
PNG
media_image5.png
263
459
media_image5.png
Greyscale
The combined VAE/GAN simultaneously learns to encode, generate, and compare dataset samples. The VAE/GAN replaces element-wise reconstruction errors with feature-wise errors for measuring reconstruction quality during training.
The claimed invention is a combination of VAE/GANs from Larsen combined with coupled GANs from Liu. The result is coupled VAE/GANs for unsupervised image-to-image translation (See Fig. 2C)
PNG
media_image6.png
673
437
media_image6.png
Greyscale
Both Liu and Larsen are directed to image to image translation among other things. Liu teaches using two sets of GANS, aka coupled generative adversarial networks (GANs), to learn a joint distribution of multi-domain images by enforcing layers to share weights in order to make a shared latent space. But Liu does not teach combining VAEs with GANs for the purpose of learning feature representations in the GAN discriminator for reconstruction instead of element-wise representations. Larsen teaches combining VAEs with GANs to learn feature-wise errors instead of element-wise errors allowing for an embedding in which high-level abstract visual features can be modified using simple arithmetic. However, Larsen does not teach learning a joint distribution of multiple domains from data. In view of the teaching of Liu, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Larsen into Liu. This would result in using VAE-GANS that are coupled to improve the quality of image translation between domains using unsupervised data.
One of ordinary skill in the art would be motivated to do this because the problems of element-wise distance metrics have been a long-standing problem in the field. (Larsen, page 6, column 2, paragraph 3, lie 1 “The problems of element-wise distance metrics are well known in the literature and many attempts have been made at going beyond pixels – typically using hand-engineered measures. Much in the spirit of deep learning, we argue that the similarity measure is yet another component which can be replaced by a learned model capable of capturing high-level structure relevant to the data distribution. In this work, our main contribution is an unsupervised scheme for learning and applying such a distance measure.”)
Regarding claim 2,
The combination of Liu and Larsen teaches the method of claim 1, wherein the generator neural network
decodes from the shared latent space, the first latent code (Larsen, Figure 1, and page 2, column 1, paragraph 3, line 1 “A VAE consists of two networks that encode a data sample x to a latent representation z and decode the latent representation back to data space, respectively:
PNG
media_image2.png
28
525
media_image2.png
Greyscale
” Examiner notes that an Encoder necessarily creates a latent space. In other words, from Figure 1, the decoder/generator network is the generator neural network that decodes from the shared latent space, and z is latent code from the shared latent space.) .
Regarding claim 4,
The combination of Liu, and Larsen teaches the method of claim 1, wherein
the generator neural network includes a generator adversarial network (Larsen, Figure 1, text “Overview of our network. We combine a VAE with a GAN by collapsing the decoder and generator into one.” In other words, from Figure 1, the generator neural network includes a GAN which is a generator adversarial network.)
Regarding claim 5,
The combination of Liu, and Larsen teaches the method of claim 1, wherein
the first and second encoder neural networks are separate encoders (Liu, Figure 1, and page 1, paragraph 1, line 1 “We propose coupled generative adversarial network (CoGAN) for learning a joint distribution of multi-domain images… It can learn a joint distribution with just samples drawn from the marginal distributions.” And, page 1, paragraph 2, line 1 “The paper concerns the problem of learning a joint distribution of multi-domain images from data.” And paragraph, 4, line 1 “CoGAN consists of a tuple of GANs, each for one image domain….We show that by enforcing a weight-sharing constraint the CoGAN can learn a joint distribution without existence of corresponding images in different domains.” In other words, GAN1 and GAN2 include a first encoder and a second encoder, each separate from the other.)
Regarding claim 7,
The combination of Liu, and Larsen teaches the method of claim 1, wherein the generator neural network includes a generator network that
uses one or more outputs of the first and second encoders to transform the one or more first images (Larsen, Figure 1. See mapping of claim 4. In other words, from Figure 1, the decoder/generator is the generator neural network which includes a generator that includes on or more outputs to transform to one or more first images.).
Regarding claim 8,
The combination of Liu, and Larsen teaches the method of claim 1, wherein the generator neural network
converts the one or more first images from a first setting to a second setting based on the shared constrained weights (Liu, Figure 1. “Each has a generative model for synthesizing realistic images in one domain and a discriminative model for classifying whether an image is real or synthesized. We tie the weights of the first few layers (responsible for decoding high-level semantics) of the generative models, g1 and g2.” And, page 1, Abstract, line 11 “We also demonstrate its applications to domain adaptation and image transformation.” In other words, synthesizing is converting, image is image, tie the weights of the first few layers is shared constrained weights of two encoders, and domain adaptation and image transformation is converting the one or more first images from a first setting to a second setting based on the shared constrained weights.).
Regarding claim 9,
The combination of Liu, and Larsen teaches the method of claim 1, wherein the generator neural network includes a generator network that is to
use the one or more features of the one or more first images and the one or more features of the one or more second images to translate the one or more first images from one setting to another setting (Liu, Figure 1. In other words, CoGAN is one or more neural networks, Generator is a generator network to use the one or more features encoded from the one or more first and second images, and domain adaptation and image transformation is to translate the one or more images from one setting to another setting. Examiner notes that “generator neural network” is previously mapped to Larsen in the mapping of claim 1.)
Regarding claim 11,
The combination of Liu, and Larsen teaches the method of claim 1, wherein
the first and second latent codes are equal (Larsen, page 2, column 1, paragraph 5, line 1 “A VAE consists of two networks that encode a data sample x to a latent representation z and decode the latent representation back to data space, respectively:
PNG
media_image7.png
25
437
media_image7.png
Greyscale
In other words, the first and second encoder neural networks generate a first and second latent code where the codes are equal.)
Regarding claim 13,
The combination of Liu, and Larsen teaches the method of claim 1, wherein the generator neural network
generates the one or more translated images by translating from a first domain to a second domain, wherein the first domain is synthetic and the second domain is real (Liu, Figure 1, “Each has a generative model for synthesizing realistic images in one domain and discriminative model for classifying whether an image is real or synthesized.” In other words, synthesizing is translating from a first domain to a second domain, synthesized is synthetic, and real is real. Examiner notes that generator neural network is previously mapped to Larsen.).
Claims 14 and 17 are a system comprising one or more processors claims that correspond to method claims 1and 4, respectively. Otherwise, they are not patentably distinct. The combination of Liu and Larsen teaches a system comprising one or more processors (Larsen, page 8, column 1, paragraph 5, line 1 “We would like to thank our reviewers for useful feedback, Søren Hauberg, Casper Kaae Sønderby and Lars Maaløe for insightful discussions, Nvidia for donating GPUs used in experiments, and the authors of DeepPy3 and CUDArray (Larsen, 2014) for the software frameworks used to implement our model.” In other words, framework is system, and GPU is one or more processors.) Therefore, claims 14 and 17 are rejected for the same reasons as claims 1 and 4, respectively.
Claim 15 is a system claim that corresponds to method claim 2. Claim 15 recites the additional limitations of “”. Liu teaches a second latent code (Liu, Figure 1. In other words, g2(z) is a second latent code.) Therefore, claim 15 is rejected for the same reasons as claim 2.
Claim 18 is a system claim that corresponds to method claim 9. Otherwise, they are not patentably distinct. Therefore, claim 18 is rejected for the same reasons as claim 9.
Claims 24 and 25 are one or more processors, comprising circuitry claims corresponding to method claims 1 and 2, respectively. The combination of Liu and Larsen teaches one or more processors comprising circuity (Larsen, page 8, column 1, paragraph 5, line 1 “We would like to thank our reviewers for useful feedback, Søren Hauberg, Casper Kaae Sønderby and Lars Maaløe for insightful discussions, Nvidia for donating GPUs used in experiments, and the authors of DeepPy3 and CUDArray (Larsen, 2014) for the software frameworks used to implement our model.” In other words, GPU is one or more processors comprising circuitry.) Otherwise, they are not patentably distinct. Therefore, claims 24 and 25 are rejected for the same reasons as claims 1 and 2, respectively.
Regarding claims 26,
The combination of Liu and Larsen teaches the method of claim 1, wherein
the equality constraint requires one or more final layer weights of the first encoder neural network to be equal to one or more final layer weights of the second encoder neural network (Liu, see mapping of claim 1 and interpretation of “equality constraint”. And, page 1, paragraph 4, line 5 “By enforcing the layers that decode high-level semantics in the GANs to share the weights, it forces the GANs to decode the high-level semantics in the same way. The layers that decode low-level details then map the shared representation to image in individual domains for confusing the respective discriminative models.” In other words, share the weights is the equality constraint requires one or more final layer weights of the first encoder neural network to be equal to one or more final layer weights of the second encoder neural network.)
Claim 27 is a system claim that corresponds to method claim 26. Otherwise, they are not patentably distinct. Therefore, claim 27 is rejected for the same reasons as claim 26.
Claim 28 is a one or more processors claim that corresponds to method claim 26. Otherwise, they are not patentably distinct. Therefore, claim 28 is rejected for the same reasons as claim 26.
Claims 3, 12, 16, and 22-23 are rejected under 35 U.S.C. § 103 as being unpatentable over Liu, Larsen, and Gatys, et al (Image Style Transfer Using Convolutional Neural Networks, herein Gatys).
Regarding claim 3,
The combination of Liu, and Larsen teaches the method of claim 1, wherein the generator neural network
Thus far, the combination of Liu and Larsen does not explicitly teach translates the one or more first images from a first domain to a second domain.
Gatys teaches translates the one or more first images from a first domain to a second domain (Gatys, Figure 7, and, page, 2420, column 2, paragraph 2, line 1 “Thus far the focus of this paper was on artistic style transfer. In general though, the algorithm can transfer the style between arbitrary images. As an example, we transfer the style of a photograph of New York by night onto an image of London in daytime (Fig 7).”
PNG
media_image8.png
491
443
media_image8.png
Greyscale
In other words, London by day is one domain and New York by night is a second domain.)
Both Gatys and the combination of Liu and Larsen are directed to image translation, among other things. The combination of Liu and Larsen teaches the computer implemented method of claim 1, but does not explicitly teach to include translating the one or more first images from a first domain to a second domain. Gatys teaches to include translating the one or more first images from a first domain to a second domain.
In view of the teaching of the combination of Liu and Larsen, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Gatys into the combination of Liu and Larsen. This would result in the computer-implemented method of claim 1, to include translating the one or more first images from a first domain to a second domain.
One of ordinary skill in the art would be motivated to do this because being able to simulate human vision would give computers the ability to compare two separate images and transform one or both. (Gatys, page 2421, column 2, paragraph 2, line 1 “Nevertheless, we find it truly fascinating that a neural system, which is trained to perform one of the core computational tasks of biological vision, automatically learns image representations that allow – at least to some extent – the separation of image content from style.”)
Regarding claim 12,
The combination of Liu, Larsen, and Gatys teaches the method of claim 1, wherein the generator neural network generates the one or more translated images by
translating from a first domain to a second domain, and wherein the first domain is day time and the second domain is night time. (Gatys, Figure 7, and page, 2420, column 2, paragraph 2, line 1 “Thus far the focus of this paper was on artistic style transfer. In general though, the algorithm can transfer the style between arbitrary images. As an example, we transfer the style of a photograph of New York by night onto an image of London in daytime (Fig 7).” In other words, London by day is day domain and New York by night is night domain.)
Claim 16 is a system claim that corresponds to method claim 3. Otherwise, they are not patentably distinct. Claim 16 recites the additional limitations of “”. The limitation of “to transform the one or more first images at least by translating the one or more first images from a first domain to a second domain” is not patentably distinct from the limitation “translating the one or more first images from a first domain to a second domain”. Therefore, claim 16 is rejected for the same reasons as claim 3.
Regarding claim 22,
The combination of Liu, Larsen, and Gatys teaches the method of claim 1, wherein the generator neural network generates the one or more translated images by
transforming the one or more first images to include the one or more features of the one or more second images (Gatys, Figure 1, and page, 2420, column 2, paragraph 2, line 1 “Thus far the focus of this paper was on artistic style transfer. In general though, the algorithm can transfer the style between arbitrary images. As an example, we transfer the style of a photograph of New York by night onto an image of London in daytime (Fig 7).” In other words, London by day is day domain and New York by night is night domain, and transferring the style of a photograph of New York by night onto an image of London in daytime is transforming the one or more first images to include the one or more features of the one or more second images.)
Regarding claim 23,
The combination of Liu, Larsen, and Gatys teaches the system of claim 14, wherein
translating the one or more first images from a first domain to a second domain, and wherein the one or more features of the one or more second images are of the second domain (Gatys, Figure 1, and page, 2420, column 2, paragraph 2, line 1 “Thus far the focus of this paper was on artistic style transfer. In general though, the algorithm can transfer the style between arbitrary images. As an example, we transfer the style of a photograph of New York by night onto an image of London in daytime (Fig 7).” In other words, London by day is first image, day is first domain, New York by night is second image, night is second domain, and transferring the style of a photograph of New York by night onto an image of London in daytime is transforming the one or more first images to include the one or more features of the second image which are from the second domain.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BART RYLANDER whose telephone number is (571)272-8359. The examiner can normally be reached Monday - Thursday 8:00 to 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at 571-270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Bart I Rylander/Examiner, Art Unit 2124
Prosecution Insights