Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed 03/25/2026. The status of the claims are as follows:
Claims 1-12 are pending
Claims 1-12 are amended
Response to Amendments
The Office action is in response to Applicant’s communication filed 03/25/2026 in response to the Office action mailed 11/25/2025. Applicant’s remarks and amendments to the claims have been considered with the results set forth below.
Upon reconsideration, the prior rejection of claims 1-12 under 35 U.S.C. § 101 is withdrawn.
Upon reconsideration, the prior rejection of claims 7-12 under 35 U.S.C. § 112(b), based on the prior “mapping unit”, “definition unit”, and “training unit” limitations, is withdrawn in view of the amendments to the system claims.
The prior rejection of claims 1-12 under 35 U.S.C. § 103 over Anirudh in view of Montserrat is withdrawn. Montserrat is not relied upon in this action.
Applicant’s amendments do not place the claims in condition for allowance because claims 1-12 remain unpatentable under 35 U.S.C. § 103 for the reasons set forth below. Applicant’s amendment necessitated the new ground(s) of rejection presented in this Office action because the amended independent claims now recite additional limitations directed to a single generator comprising residual connections and configured to perform upsampling operations, a perceptual loss measuring a difference in features of a Visual Geometry Group (VGG) model, regularization loss limiting latent vectors to a present target distribution, and operation within a discriminator network, adversarial loss, or encoder network. Accordingly, this action is made final.
Response to Arguments
Regarding 35 U.S.C. § 101
The Examiner has considered Applicant’s arguments and amendments regarding the rejection of claims 1-12 under 35 U.S.C. § 101 and finds them persuasive.
Applicant argues that the amended claims are directed to a specific technical solution for training a self-converging generative network for image generation, including generating an output image using a single generator and performing a training process that updates both generator weight parameters and latent vectors such that the latent vectors define a latent space aligned with a present target distribution. Applicant further argues that the amended claims specify that the training process operates without using a discriminator network, adversarial loss, or encoder network.
Upon reconsideration, the Examiner agrees that the amended claims now recite a more particular ordered combination directed to a specific generative-network training process rather than merely reciting model training at a high level of generality.
Accordingly, the rejection of claims 1-12 under 35 U.S.C. § 101 is withdrawn.
Regarding 35 U.S.C. § 112
The Examiner has considered Applicant’s arguments and amendments regarding the rejection of claims 7-12 under 35 U.S.C. § 112(b) and finds them persuasive with respect to the prior rejection.
Applicant argues that the amendments regarding the rejection of claims 7-12 overcome the prior §112(b) issues concerning the prior “mapping unit”, “definition unit”, and “training unit” limitations. These arguments are persuasive with respect to the prior §112(b) rejection because the amended system claims now recite one or more processors and memory storing instructions which, when executed, cause the system to perform the recited operations.
Accordingly, the prior rejection of claims 7-12 under 35 U.S.C. § 112(b) is withdrawn.
Regarding 35 U.S.C. § 103
The Examiner has considered Applicant’s arguments and amendments regarding the rejection of claims 1-12 under 35 U.S.C. § 103 and finds them persuasive to the extent that the prior rejection over Anirudh in view of Montserrat is withdrawn. However, Applicant’s arguments are not persuasive of patentability over the newly cited combination set forth below.
Applicant argues that Anirudh and Montserrat do not disclose or suggest a training framework in which a single generator comprising residual connections and upsampling is trained using a generator loss including image loss, perceptual loss, and regularization loss, while a latent space is trained using a latent-space loss that includes both image loss and regularization loss. Applicant further argues that the cited references do not disclose or suggest operation without a discriminator network, adversarial loss, or encoder network.
These arguments are not persuasive against the present rejection.
As an initial matter, the prior rejection over Anirudh in view of Montserrat is withdrawn, and Montserrat is not relied upon in this action. The present rejection relies on Hoshen, Gao, and Kingma to address the amended limitations. Hoshen teaches an encoder-less, non-adversarial generative image model in which training images are embedded using latent vectors, a generator maps latent vectors to images, generator weights and latent vectors are optimized, and a VGG perceptual loss is used instead of a discriminator. Gao teaches a generator architecture comprising residual-network modules and multiscale upsampling components. Kingma teaches latent-variable regularization using a KL-divergence term that encourages latent variables to conform to a prior distribution, such as a centered isotropic multivariate Gaussian.
Applicant’s argument that the previous references did not teach residual connections, upsampling operations, VGG perceptual loss, and the claimed negative limitations is therefore not persuasive against the present combination because those limitations are addressed by Hoshen and Gao as set forth below.
Applicant’s arguments regarding Montserrat is moot because Montserrat is not relied upon in this action.
Applicant’s argument that a reference using adversarial training cannot teach the claimed invention is not persuasive against the present rejection. Gao is relied upon only for the generator architecture comprising residual connections and upsampling operations, not for Gao’s discriminator or adversarial-training framework. Hoshen teaches that the training process may be performed without a discriminator and without adversarial loss, and the proposed combination preserves Hoshen’s non-adversarial training framework while using Gao’s known generator architecture features.
Applicant’s arguments that an encoder-based reference cannot teach the claimed invention is also not persuasive against the present rejection. Kingma is relied upon only for the known latent-variable regularization technique of using a divergence term to encourage latent variables to conform to a prior distribution. Kingma is not relied upon for incorporating an encoder network into Hoshen. Hoshen already teaches directly optimizing latent vectors associated with training images; therefore, applying Kingma’s latent-variable regularization teaching to Hoshen’s latent vectors would not require adding an encoder network.
Accordingly, the rejection of claims 1-12 under 35 U.S.C. § 103 is newly set forth in modified form below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, and 7-11 are rejected under 35 U.S.C. 103 as being unpatentable over Hoshen et al. ("Non-Adversarial Image Synthesis with Generative Latent Nearest Neighbors") in view of Gao et al. (US11403735B2) and further in view of Kingma et al. ("Auto-Encoding Variational Bayes").
Regarding claim 1, Hoshen in view of Gao and further in view of Kingma, teach a computer-implemented method for training a self-converging generative network, the method comprising:
“associating each training image of a training dataset with a respective latent vector representing the training image in a latent space;” – Hoshen teaches this limitation. Hoshen discloses Generative Latent Optimization (GLO), in which training images are embedded in a low-dimensional latent space:
“GLO, introduced by Bojanowski et al., embeds the training images in a low dimensional space, so that they are reconstructed when the embedding is passed through a jointly trained deep generator.” (Hoshen, § 1. Introduction)
Thus, Hoshen teaches associating each training image
x
i
of a training dataset with a respective latent vector
z
i
representing the training image in a latent space.
“generating, using a single generator of the self-converging generative network, an output image from the respective latent vector, ” – Hoshen teaches this limitation. Hoshen teaches generating, using a generator G, an output image from the respective latent vector:
“The latent code is projected by the generator to yield image
I
=
G
(
z
)
” (Hoshen, Figure 1)
Hoshen further discloses that GLO replaces a linear matrix with:
“a deep CNN generator
G
(
)
which is more suitable for modeling images.” (Hoshen, § 3.1. GLO)
Hoshen also discloses that, during sampling:
“We can therefore use the generator to project the latent code to image space by our GLO trained generator G():
I
e
=
G
(
z
e
)
” (Hoshen, § 3.4.3)
“iteratively updating weight parameters of the single generator and the latent vectors during a training process” — Hoshen teaches this limitation. Hoshen discloses the GLO optimization objective:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen further discloses:
“All weights are trained by SGD (including the generator weights
G
(
)
and a latent vector
z
i
per each training image
x
i
). After training, the result is a generator
G
(
)
and a latent embedding
z
i
of each training image
x
i
.” (Hoshen, §3.1. GLO)
Hoshen further discloses, with respect to the VGG perceptual-loss objective:
“All parameters are optimized directly by SGD.” (Hoshen, p. 4)
Thus, Hoshen teaches iteratively updating generator weight parameters and latent vectors during a training process.
“such that the latent vectors define the latent space ” — Hoshen teaches or suggests this limitation at least in part. Hoshen teaches constraining learned latent vectors in a latent space. Hoshen discloses that GLO differs from other methods by:
“Constraining all latent vectors to lie on a unit sphere or a unit ball.” (Hoshen, §3.1. GLO)
Hoshen also teaches mapping between an arbitrary distribution and the GLO latent space. Hoshen discloses that GLANN includes:
“Mapping between an arbitrary distribution (typically a multi-dimensional normal distribution) and the low-dimensional latent space using IMLE.” (Hoshen, §3.4. GLANN: Generative Latent Nearest Neighbor)
“the training process comprising: (i) calculating a loss function of the single generator comprising: (a) an image loss measuring a difference between the output image and the corresponding training image” — Hoshen teaches this limitation. Hoshen discloses the GLO objective:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
In this objective,
G
(
z
i
)
is the image generated from latent vector
z
i
, and
x
i
is the corresponding training image. Therefore, the loss
l
(
G
(
z
i
)
,
x
i
)
measures a difference between the generated output image and the corresponding training image.
Hoshen further discloses:
“The latent code is projected by the generator to yield image
I
=
G
(
z
)
.” (Hoshen, Figure. 1)
Thus, Hoshen teaches calculating an image loss measuring a difference between the output image generated by the generator and the corresponding training image.
“(b) a perceptual loss measuring a difference in features of a Visual Geometry Group (VGG) model between the output image and the corresponding training image” — Hoshen teaches this limitation. Hoshen discloses:
“Differently from the GLO algorithm, we use a VGG perceptual loss function.” (Hoshen, p. 4)
Hoshen further discloses the perceptual-loss optimization objective:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen further discloses:
“We do not have a discriminator, instead, we use a VGG perceptual loss.” (Hoshen, §4.1. Quantitative Image Generation Results)
Hoshen further discloses:
“In this work, we replaced the standard adversarial loss function by a perceptual loss.” (Hoshen, §5. Discussion)
Thus, Hoshen teaches a perceptual loss measuring a difference in VGG feature space between the generated output image and the corresponding training image.
“and (c) a regularization loss limiting the latent vectors ” — Hoshen teaches this limitation in part. Hoshen teaches constraining learned latent vectors. Hoshen discloses:
“Constraining all latent vectors to lie on a unit sphere or a unit ball.” (Hoshen, §3.1. GLO)
Hoshen further discloses a constrained optimization objective in which the latent vector zi is constrained according to:
“
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
“and (ii) calculating a loss function of the latent space ” — Hoshen teaches this limitation in part. Hoshen teaches optimizing latent vectors
z
i
using a loss comparing generated images
G
(
z
i
)
to corresponding training images xi. Hoshen discloses:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen further discloses:
“All weights are trained by SGD (including the generator weights
G
(
)
and a latent vector
z
i
per each training image
x
i
). After training, the result is a generator
G
(
)
and a latent embedding
z
i
of each training image
x
i
.” (Hoshen, §3.1. GLO)
Thus, Hoshen teaches calculating a loss used to update latent vectors, wherein the loss includes an image/perceptual image loss comparing the generated output image and corresponding training image.
“wherein the training process operates without using a discriminator network, adversarial loss, or encoder network” — Hoshen teaches this limitation. Hoshen discloses that GLO is:
“a recently introduced encoder-less generative model which uses a non-adversarial loss function.” (Hoshen, §2. Previous Work)
Hoshen further discloses:
“We do not have a discriminator, instead, we use a VGG perceptual loss.” (Hoshen, §4.1. Quantitative Image Generation Results)
Hoshen further discloses:
“In this work, we replaced the standard adversarial loss function by a perceptual loss.” (Hoshen, §5. Discussion)
Thus, Hoshen teaches a training process operating without using an encoder network, discriminator network, or adversarial loss.
Hoshen does not teach these limitations and/or portions of:
“the single generator comprising residual connections and configured to perform upsampling operations;”
“aligned with a preset target distribution;”
“to the preset target distribution;” and
“comprising the image loss and the regularization loss.”
Gao, however, teaches these limitations and/or portions of:
“the single generator comprising residual connections and configured to perform upsampling operations” — Gao teaches a generator comprising a residual-network module and a multiscale upsampling component. Gao discloses:
“The generator model G 610 is composed of two components, the residual network module 612 and the multiscale upsampling component 614.” (Gao, col. 10, lines 19-21)
Gao further discloses that the residual-network module includes residual blocks:
“The core of the residual network module 612, the residual network building block 720, is shown in FIG. 7.” (Gao, col. 10, lines 22-23)
Gao further discloses:
“Instead of using a convolutional layer to directly fit the transformation between the input feature map and the output feature map, the residual block 720 tries to fit the residue of the output deduced by the input.” (Gao, col. 10, lines 23-27)
Gao further discloses that the outputs of a shortcut block and residual block are combined:
“The outputs of these two components 706 and 720 are then added together elementwise at adder 708. In this implementation, the residual network module 612 consists of 16 residual blocks 720.” (Gao, col. 10, lines 46-49)
Thus, Gao teaches a generator comprising residual connections through its residual-network module, residual blocks, shortcut path, and elementwise addition of the shortcut and residual-block outputs.
Gao further teaches that the generator is configured to perform upsampling operations. Gao discloses:
“The output from this layer is then fed to the multiple multiscale upsampling component 614.” (Gao, col. 10, lines 64-66)
Gao further discloses:
“After this feature map extraction process, the multiscale upsampling component 614 uses pixel shuffle layers 730, 732, and 734 combined with the convolutional layers 740 and 742 to gradually increase the dimensionality of the input image.” (Gao, col. 10-11, lines 66-3)
Gao further discloses:
“The multiscale upsampling component 614, which eliminates the fake details, is composed of several pixel shuffle layers 730, 732 and 734 and plural convolutional layers 740 and 742. Using these layers, the model of FIGS. 6 and 7 is able to process 2x, 4x, and 8x super-resolution images.” (Gao, col. 11, lines 4-8)
Gao further discloses:
“The multiscale upsampling component 614 includes pixel shuffle layers (PSX2) 730, 732, and 734, and convolutional layers 740 and 742, linked as shown in FIG. 7. This means that the pixel shuffle layers 730, 732, and 734, whose scaling factor is 2, and which is used to perform the upscaling of the figure dimensionality, is capable of outputting 2x, 4x, and 8x high-resolution images 750, 752, and 754.” (Gao, col. 11, lines 27-33)
Thus, Gao teaches a single generator comprising residual connections and configured to perform upsampling operations.
It would have been obvious to modify Hoshen’s generator to include Gao’s residual-network module and multiscale upsampling component because Gao teaches that the residual architecture is more effective than a traditional convolutional architecture and addresses model degradation and gradient explosion or vanishing, while the multiscale upsampling component gradually increases image dimensionality and generates higher-resolution images. A person of ordinary skill in the art would have had a reasonable expectation of success because Hoshen and Gao both concern neural-network generators for producing images, and Gao’s residual and upsampling components constitute known generator-architecture features that could have been incorporated into Hoshen’s generator while retaining Hoshen’s encoder-less, discriminator-free, and non-adversarial training framework.
Neither Hoshen nor Gao teach these remaining limitations and/or portions of:
“aligned with a preset target distribution”
“to the preset target distribution”
“comprising the image loss and the regularization loss”
Kinga, however, teaches these limitations and/or portions of:
“aligned with a preset target distribution” – Kingma teaches a latent-variable generative model including a prior distribution over latent variables. Kingma discloses:
“Let the prior over the latent variables be the centered isotropic multivariate Gaussian
p
θ
z
=
N
(
z
;
0
,
I
)
” (Kingma, pg. 5, § 3 Example: Variational Auto-Encoder)
Kingma further discloses a variational objective including a KL-divergence term between an approximate latent-variable posterior and the prior distribution:
PNG
media_image1.png
41
568
media_image1.png
Greyscale
(Kingma, pg. 3, Eq. 3)
Kingma further discloses:
“The KL-divergence term can then be interpreted as regularizing φ, encouraging the approximate posterior to be close to the prior
p
θ
(
z
)
.” (Kingma, pg. 4, §2.3)
Thus, Kingma teaches regularizing latent variables toward a preset prior distribution, such as a centered isotropic multivariate Gaussian.
It would have been obvious to apply Kingma’s latent-variable regularization teaching to Hoshen’s learned latent vectors because Hoshen already teaches directly optimizing latent vectors associated with training images and recognizes the need for a well-behaved latent space for sampling. A person of ordinary skill in the art would have had reason to regularize Hoshen’s latent vectors toward a preset target distribution, such as a normal distribution, to improve latent-space organization and sampling behavior in the generative image model.
“to the preset target distribution” – Kingma teaches a KL-divergence regularization term that encourages latent variables to be close to a prior distribution. Kingma discloses:
“The KL-divergence term can then be interpreted as regularizing φ, encouraging the approximate posterior to be close to the prior
p
θ
(
z
)
.” (Kingma, pg. 4, §2.3)
Kingma further teaches that the prior may be a centered isotropic multivariate Gaussian:
“Let the prior over the latent variables be the centered isotropic multivariate Gaussian
p
θ
z
=
N
(
z
;
0
,
I
)
” (Kingma, pg. 5, § 3 Example: Variational Auto-Encoder)
Thus, Kingma teaches a regularization loss limiting or encouraging latent variables toward a preset target distribution.
It would have been obvious to include Kingma’s KL-divergence latent regularization in Hoshen’s generator/latent training objective because Hoshen already optimizes latent vectors and generator weights, and because Kingma teaches that the KL-divergence term regularizes latent variables by encouraging them to be close to a prior distribution. A person of ordinary skill in the art would have had reason to add this known latent-space regularization to Hoshen’s learned latent vectors to improve latent-space organization and sampling from the generative model.
“comprising the image loss and the regularization loss” – Kingma teaches adding a latent-space regularization term to a reconstruction objective. Kingma discloses:
PNG
media_image1.png
41
568
media_image1.png
Greyscale
(Kingma, pg. 3, Eq. 3)
Kingma further discloses:
“The KL-divergence term can then be interpreted as regularizing φ, encouraging the approximate posterior to be close to the prior
p
θ
(
z
)
.” (Kingma, pg. 4, §2.3)
Thus, Hoshen teaches a latent-vector loss including image/perceptual reconstruction loss, and Kingma teaches adding a latent-variable regularization loss toward a preset prior distribution.
It would have been obvious to calculate a latent-space loss for Hoshen’s latent vectors that includes both the image loss taught by Hoshen and the regularization loss taught by Kingma because the combined loss would predictably preserve reconstruction fidelity while encouraging the latent vectors to conform to a structured preset distribution. A person of ordinary skill in the art would have understood that such a combined loss would improve latent-space organization and sampling behavior in a generative model.
It would have been obvious to modify Hoshen’s generator to include Gao’s residual-network module and multiscale upsampling component because Gao teaches that residual architecture improves model effectiveness and addresses model degradation and gradient explosion/vanishing problems, and that multiscale upsampling gradually increases image dimensionality for high-quality image-generation processing. A person of ordinary skill in the art would have had a reasonable expectation of success because Hoshen and Gao both concern neural-network image-generation/reconstruction models, and Gao’s residual and upsampling components are generator-architecture features that could be applied to Hoshen’s generator without changing Hoshen’s non-adversarial training framework.
It would have been further obvious to apply Kingma’s latent-variable regularization teaching to Hoshen’s learned latent vectors because Hoshen teaches directly optimizing latent vectors associated with training images and recognizes the importance of a well-behaved latent space for sampling. Kingma teaches that a KL-divergence term can regularize latent variables by encouraging them to be close to a prior distribution, such as a centered isotropic multivariate Gaussian. A person of ordinary skill in the art would have had reason to combine these teachings to improve latent-space organization and sampling behavior while preserving Hoshen’s encoder-less, non-adversarial generator training.
Regarding claim 2, Hoshen in view of Gao and further in view of Kingma teaches the method of claim 1, wherein the training process comprises:
“training the latent vectors — Hoshen teaches this limitation in part. Hoshen teaches training latent vectors associated with training images using a loss function based on a difference between the generated output image and the corresponding training image. Hoshen discloses the GLO optimization objective:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen further discloses that GLO uses a generator and latent vectors, and that:
“All weights are trained by SGD (including the generator weights
G
(
)
and a latent vector
z
i
per each training image
x
i
). After training, the result is a generator
G
(
)
and a latent embedding
z
i
of each training image
x
i
.” (Hoshen, §3.1. GLO)
Thus, Hoshen teaches training latent vectors using a loss function derived from a difference between the generated output image
G
(
z
i
)
and corresponding training image
x
i
. This corresponds to an image/pixel reconstruction loss because the loss compares generated image data to training image data.
Hoshen further teaches using normally distributed latent variables in connection with the latent space. Hoshen discloses that GLANN maps between:
“an arbitrary distribution (typically a multi-dimensional normal distribution) and the low-dimensional latent space using IMLE.” (Hoshen, p. 4)
Hoshen further discloses that, for sampling new images:
“We first sample a noise vector from the multivariate normal distribution
e
~
N
(
0
,
I
)
.” (Hoshen, p. 4)
Hoshen, does not teach these limitations and/or portions of:
“to follow a normal distribution using a loss function derived from a pixel-wise loss and Kullback-Leibler (KL) divergence” – Hoshen does not expressly teach training the latent vectors to follow a normal distribution using a loss function derived from both pixel-wise loss and KL divergence.
Kingma, however, teaches these remaining limitations and/or portions of:
“to follow a normal distribution using a loss function derived from a pixel-wise loss and Kullback-Leibler (KL) divergence” – Kingma teaches using KL divergence as a regularization term to train latent variables toward a normal prior distribution. Kingma discloses:
“Let the prior over the latent variables be the centered isotropic multivariate Gaussian
p
θ
z
=
N
(
z
;
0
,
I
)
” (Kingma, pg. 5, § 3 Example: Variational Auto-Encoder)
Kingma further discloses that the training objective includes a KL-divergence term:
“The KL-divergence
D
K
L
(
q
∅
(
z
|
x
i
)
|
|
p
θ
z
)
” (Kingma, pg. 3, Eq. 3)
Kingma further discloses:
“The KL-divergence term can then be interpreted as regularizing φ, encouraging the approximate posterior to be close to the prior
p
θ
(
z
)
.” (Kingma, pg. 4, §2.3)
Kingma also teaches that the objective includes both a KL-divergence regularization term and a reconstruction-error term. Kingma discloses:
“The first term is (the KL divergence of the approximate posterior from the prior) acts as a regularizer, while the second term is a an expected negative reconstruction error.” (Kingma, pg. 4, §2.3)
Thus, Kingma teaches training latent variables to follow a normal prior distribution using a loss function derived from reconstruction error and KL divergence.
It would have been obvious to modify Hoshen’s latent-vector training to include Kingma’s KL-divergence regularization toward a centered isotropic multivariate Gaussian because Hoshen already teaches training latent vectors associated with training images and recognizes the desirability of mapping a normal distribution to the latent space for image generation. Kingma teaches the known technique of using KL divergence as a regularization term to encourage latent variables to conform to a normal prior distribution while also using a reconstruction-error term. A person of ordinary skill in the art would have had reason to combine Hoshen’s image reconstruction/perceptual training of latent vectors with Kingma’s KL-divergence latent regularization to obtain latent vectors that both reconstruct corresponding training images and follow a normal distribution for improved latent-space organization and sampling.
Regarding claim 3, Hoshen in view of Gao and further in view of Kingma teaches the method of claim 2, wherein
“the latent space is trained to self-converge ” – Hoshen teaches this limitation in part. As discussed above with respect to claims 1 and 2, Hoshen teaches training latent vectors associated with respective training images while jointly training a generator. Hoshen discloses:
“GLO, introduced by Bojanowski et al., embeds the training images in a low dimensional space, so that they are reconstructed when the embedding is passed through a jointly trained deep generator.” (Hoshen, §1. Introduction)
Hoshen further discloses:
“The resulting
z
i
are latent vectors that embed the data in a lower dimension and typically better behaved space.” (Hoshen, §3.1. GLO)
Hoshen further discloses:
“All weights are trained by SGD (including the generator weights
G
(
)
and a latent vector
z
i
per each training image
x
i
). After training, the result is a generator
G
(
)
and a latent embedding
z
i
of each training image
x
i
.” (Hoshen, §3.1. GLO)
Thus, Hoshen teaches training the latent vectors and generator such that the latent vectors form a learned latent space associated with the training images. This corresponds to training the latent space to converge during the training process because the latent vectors are optimized by SGD together with the generator until the training objective is minimized.
Hoshen further teaches constraining the latent vectors during training. Hoshen discloses that GLO differs from other methods by:
“Constraining all latent vectors to lie on a unit sphere or a unit ball.” (Hoshen, §3.1. GLO)
Hoshen further discloses the constrained GLO objective:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen also discloses use of a normal distribution in connection with the latent space. Hoshen discloses that GLANN includes:
“Mapping between an arbitrary distribution (typically a multi-dimensional normal distribution) and the low-dimensional latent space using IMLE.” (Hoshen, §3.4. GLANN: Generative Latent Nearest Neighbor)
Hoshen further discloses:
“We first sample a noise vector from the multivariate normal distribution
e
~
N
(
0
,
I
)
.” (Hoshen, §3.4.3 Sampling new images)
Hoshen does not teach these limitations and/or portions of:
“... and follow a normal distribution during the training process.” – Hoshen does not expressly teach that the latent space is trained to follow a normal distribution during the training process using KL-divergence regularization.
Kingma, however, teaches these remaining limitations and/or portions of:
“... and follow a normal distribution during the training process.” – Kingma teaches this portion. Kingma discloses a latent-variable generative model in which the prior over latent variables is a normal distribution:
“Let the prior over the latent variables be the centered isotropic multivariate Gaussian
p
θ
z
=
N
(
z
;
0
,
I
)
” (Kingma, pg. 5, §3 Example: Variational Auto-Encoder)
Kingma further discloses a training objective including a KL-divergence term:
“The KL-divergence
D
K
L
(
q
∅
(
z
|
x
i
)
|
|
p
θ
z
)
” (Kingma, pg. 3, Eq. 3)
Kingma further discloses:
“The KL-divergence term can then be interpreted as regularizing φ, encouraging the approximate posterior to be close to the prior
p
θ
(
z
)
.” (Kingma, pg. 4, §2.3)
Thus, Kingma teaches training latent variables such that the latent-variable distribution is encouraged to follow a preset normal distribution during training.
It would have been obvious to modify Hoshen’s latent-vector training to include Kingma’s KL-divergence regularization toward a centered isotropic multivariate Gaussian because Hoshen already trains latent vectors associated with training images and teaches sampling from a multivariate normal distribution for image generation. Kingma teaches that KL-divergence regularization encourages latent variables to be close to a normal prior distribution. A person of ordinary skill in the art would have had reason to apply Kingma’s known latent-variable regularization to Hoshen’s learned latent vectors so that Hoshen’s latent space would converge during training while being organized according to a normal distribution, thereby improving latent-space structure and sampling behavior.
Regarding claim 4, Hoshen in view of Gao and further in view of Kingma teaches the method of claim 1, wherein
“the latent vectors are – Hoshen teaches this limitation in part. Hoshen teaches associating each training image with a respective trainable latent vector. Hoshen discloses:
“All weights are trained by SGD (including the generator weights
G
(
)
and a latent vector
z
i
per each training image
x
i
). After training, the result is a generator
G
(
)
and a latent embedding
z
i
of each training image
x
i
.” (Hoshen, §3.1. GLO)
Thus, Hoshen teaches a one-to-one association between each training image xi and a corresponding latent vector zi.
Hoshen further teaches the use of randomly sampled normally distributed vectors in connection with its latent-space image-generation process. Hoshen discloses that the GLANN architecture begins with:
“a random noise vector is sampled and mapped to the latent space to yield latent code
z
=
T
(
e
)
.
” (Hoshen, Figure 1)
Hoshen further discloses that IMLE includes:
“
M
random latent codes
e
j
[that] are sampled from a normal distribution.” (Hoshen, §3.2. IMLE)
Hoshen also discloses that GLANN maps between:
“an arbitrary distribution (typically a multi-dimensional normal distribution) and the low-dimensional latent space using IMLE.” (Hoshen, §3.4. GLANN: Generative Latent Nearest Neighbor)
Hoshen further discloses:
“We first sample a noise vector from the multivariate normal distribution
e
~
N
(
0
,
I
)
.” (Hoshen, §3.4.3 Sampling new images)
Thus, Hoshen teaches randomly sampling vectors from a normal distribution for use in forming latent codes and generating images.
Hoshen does not teach these limitations and/or portions of:
“... randomly initialized using a normal distribution ...” – Hoshen does not expressly teach that each trainable latent vector associated with a respective training image is initially sampled from a normal distribution having a preset standard deviation.
Kingma, however, teaches these remaining limitations and/or portions of:
“... randomly initialized using a normal distribution ...” – Kingma teaches randomly sampling latent variables from a Gaussian distribution having a specified standard deviation. Kingma discloses, for a Gaussian latent variable:
“let
z
~
p
z
x
=
N
(
μ
,
σ
2
)
. In this case, a valid reparameterization is z = µ + σϵ, where ϵ is an auxiliary noise variable ϵ ~
N
(0, 1).” (Kingma, pg. 5, §2.4 The reparameterization trick)
Kingma further explains that, for location-scale distributions:
“we can choose the standard distribution (with location = 0, scale = 1) as the auxiliary variable ϵ, and let g(.) = location + scale · ϵ.” (Kingma, pg. 5, §2.4 The reparameterization trick)
Kingma further discloses a centered isotropic multivariate Gaussian prior:
“Let the prior over the latent variables be the centered isotropic multivariate Gaussian
p
θ
z
=
N
(
z
;
0
,
I
)
” (Kingma, pg. 5, §3 Example: Variational Auto-Encoder)
Kingma also teaches sampling a latent variable according to a Gaussian distribution having a defined mean and standard deviation:
“
z
(
i
,
l
)
~
q
∅
z
x
i
u
s
i
n
g
z
(
i
,
l
)
=
g
∅
x
i
ε
l
=
μ
(
i
)
+
σ
(
i
)
⨀
ε
l
w
h
e
r
e
ϵ
l
~
N
(
0
,
I
)
” (Kingma, §3. Example: Variational Auto-Encoder)
Thus, Kingma teaches randomly sampling latent variables from a normal distribution using a specified scale or standard deviation.
It would have been obvious to initialize Hoshen’s respective trainable latent vectors using Kingma’s known Gaussian sampling technique because Hoshen requires an initial latent vector
z
i
for each training image xi before jointly optimizing the latent vectors and generator weights. Kingma teaches that latent variables may be generated by sampling from a normal distribution having a specified mean and standard deviation. A person of ordinary skill in the art would have had reason to initialize Hoshen’s per-image latent vectors by randomly sampling from a normal distribution having a preset standard deviation because such initialization provides defined initial values for gradient-based optimization, is consistent with Hoshen’s use of normally distributed latent inputs, and facilitates organizing the learned latent space relative to a normal target distribution. The resulting method would retain Hoshen’s one-to-one pairing of each latent vector
z
i
with its corresponding training image
x
i
while initializing each latent vector from a normal distribution having a preset standard deviation.
Regarding claim 5, Hoshen in view of Gao and further in view of Kingma teaches the method of claim 1, wherein:
“the training of the self-converging generative network is performed based on the loss function of the single generator and the loss function of the latent space” — Hoshen teaches or suggests this limitation. Hoshen teaches jointly training the generator and the latent vectors based on a loss comparing an image generated from a latent vector with the corresponding training image. Hoshen discloses the GLO optimization objective:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen further discloses:
“All weights are trained by SGD (including the generator weights
G
(
)
and a latent vector
z
i
per each training image
x
i
). After training, the result is a generator
G
(
)
and a latent embedding
z
i
of each training image
x
i
.” (Hoshen, §3.1. GLO)
Hoshen further teaches a VGG perceptual-loss objective:
PNG
media_image2.png
73
474
media_image2.png
Greyscale
(Hoshen, Eq. 3, §3.4.1 Stage 1: Latent embedding)
Hoshen further discloses:
“All parameters are optimized directly by SGD.” (Hoshen, §3.4.1 Stage 1: Latent embedding)
Thus, Hoshen teaches performing training based on a loss function that is used to update both the single generator
G
and the latent vectors
z
i
. The loss functions applied to the generator and latent vectors correspond respectively to the claimed loss function of the single generator and loss function of the latent space.
“wherein the loss function of the single generator acquires a relationship between the latent space and an image space” — Hoshen teaches this limitation. Hoshen teaches that the generator is jointly trained with latent vectors so that the generator maps the latent-space representations of training images into corresponding images in image space. Hoshen discloses:
“GLO, introduced by Bojanowski et al., embeds the training images in a low dimensional space, so that they are reconstructed when the embedding is passed through a jointly trained deep generator.” (Hoshen, §3.4.1 Stage 1: Latent embedding)
Hoshen further discloses:
“The latent code is projected by the generator to yield image
I
=
G
(
z
)
.” (Hoshen, Figure. 1)
Hoshen further discloses that GLANN includes:
“embedding the high-dimensional image space into a ‘well-behaved’ latent space using GLO.” (Hoshen, §3.4. GLANN: Generative Latent Nearest Neighbor)
Hoshen further teaches training the generator and latent vectors using the perceptual-loss objective:
PNG
media_image2.png
73
474
media_image2.png
Greyscale
(Hoshen, Eq. 3, §3.4.1 Stage 1: Latent embedding)
Hoshen further discloses:
“By the end of training, the training images are embedded by the low dimensional latent codes
{
z
i
}
.” (Hoshen, §3.4.1 Stage 1: Latent embedding)
Hoshen further discloses that the trained generator projects a latent code into image space:
“We can therefore use the generator to project the latent code to image space by our GLO trained generator
G
(
)
:
I
e
=
G
(
z
e
)
.” (Hoshen, §3.4.3. Sampling new images)
Thus, by minimizing the generator loss between generated images
G
(
z
i
)
and corresponding training images
x
i
, Hoshen’s generator learns or acquires a relationship mapping the latent space represented by latent vectors
z
i
to the image space represented by generated images
G
(
z
i
)
.
“and the loss function of the latent space limits the latent vectors to the preset target distribution — Hoshen teaches this limitation in part. Hoshen teaches limiting latent vectors during training by constraining the latent vectors to a defined latent-space region. Hoshen discloses:
“Constraining all latent vectors to lie on a unit sphere or a unit ball.” (Hoshen, §3.1. GLO)
Hoshen further discloses the constraint:
“
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen also recognizes the use of a normal distribution in connection with its learned latent space. Hoshen discloses that GLANN includes:
“Mapping between an arbitrary distribution (typically a multi-dimensional normal distribution) and the low-dimensional latent space using IMLE.” (Hoshen, §3.4. GLANN: Generative Latent Nearest Neighbor)
Hoshen does not teach these limitations and/or portions of:
“... using KL divergence” – However, Hoshen does not expressly teach limiting the latent vectors to a preset target distribution using KL divergence.
Kingma, however, teaches these limitations and/or portions of:
“... using KL divergence” – Kingma teaches a training objective comprising a KL-divergence term between a learned latent-variable distribution and a preset prior distribution. Kingma discloses:
“Let the prior over the latent variables be the centered isotropic multivariate Gaussian
p
θ
z
=
N
(
z
;
0
,
I
)
” (Kingma, pg. 5, §3 Example: Variational Auto-Encoder)
Kingma further discloses:
“The KL-divergence
D
K
L
(
q
∅
(
z
|
x
i
)
|
|
p
θ
z
)
” (Kingma, pg. 3, Eq. 3)
Kingma explains:
“The KL-divergence term can then be interpreted as regularizing φ, encouraging the approximate posterior to be close to the prior
p
θ
(
z
)
.” (Kingma, pg. 4, §2.3)
Kingma further explains that its training objective includes both the KL-divergence regularizer and a reconstruction term:
“The first term is (the KL divergence of the approximate posterior from the prior) acts as a regularizer, while the second term is a an expected negative reconstruction error.” (Kingma, pg. 4, §2.3)
Thus, Kingma teaches a latent-space loss using KL divergence to limit or encourage latent variables toward a preset target distribution, such as a centered isotropic multivariate Gaussian.
It would have been obvious to include Kingma’s KL-divergence regularization term in the loss used to train Hoshen’s latent vectors because Hoshen already teaches directly optimizing a respective latent vector for each training image and recognizes the desirability of a structured latent space associated with a normal distribution. Kingma teaches that KL divergence provides a known loss term for encouraging latent variables to conform to a preset prior distribution. A person of ordinary skill in the art would have had reason to apply Kingma’s KL-divergence regularization to Hoshen’s latent-vector loss to limit the learned latent vectors toward a preset target distribution while Hoshen’s generator loss learns the relationship between the latent space and image space.
Kingma is relied upon only for the known use of KL-divergence regularization to align latent variables with a preset prior distribution. Kingma is not relied upon for incorporating an encoder network into Hoshen. Hoshen already teaches directly optimizing the latent vectors associated with respective training images, and application of Kingma’s KL-divergence loss to those trainable latent vectors would not require adding an encoder network.
Claims 6 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Hoshen in view of Gao further in view of Kingma and further in view of Anirudh et al. (US11126895B2).
Regarding claim 6, Hoshen in view of Gao and Kingma, and further in view of Anirudh teaches the method of claim 1, wherein:
“the training process comprises — Hoshen teaches jointly training the generator weight parameters and latent vectors. Hoshen discloses the GLO optimization objective:
“
a
r
g
G
m
i
n
{
z
i
}
∑
i
l
G
z
i
,
x
i
s
.
t
.
z
i
=
1
” (Hoshen, Eq. 2)
Hoshen further discloses:
“All weights are trained by SGD (including the generator weights
G
(
)
and a latent vector
z
i
per each training image
x
i
). After training, the result is a generator
G
(
)
and a latent embedding
z
i
of each training image
x
i
.” (Hoshen, §3.1. GLO)
Hoshen further teaches, with respect to its perceptual-loss objective:
“All parameters are optimized directly by SGD.” (Hoshen, § 3.4.1 Stage 1: Latent embedding)
Thus, Hoshen teaches training both the generator weight parameters and the latent vectors during the training process.
Hoshen also recognizes alternating optimization as a known technique for jointly optimizing model parameters and latent vectors. In discussing latent-factor optimization, Hoshen discloses:
“Both W and zi are optimized directly e.g. by alternating least squares or SVD.” (Hoshen, §3.1. GLO)
Hoshen does not teach these limitations and/or portions of:
“... alternately ...” – Hoshen does not expressly teach that the generator weight parameters and latent vectors of the GLO generator are trained in alternating phases.
Anirudh, however, teaches these limitations and/or portions of:
“... alternately ...” – Anirudh teaches a training procedure in which neural-network parameters and latent vectors are updated in separate phases. Anirudh discloses:
“The CM system uses the update CMN component to update parameters θ of the CMN by applying a gradient descent technique based on the loss function.” (Anirudh, col. 3, lines 45-47)
Anirudh then separately updates the latent vectors:
“To learn the latent vectors, the CM system iteratively applies a gradient descent technique to update the latent vectors based on the loss function.” (Anirudh, col. 3, lines 2-5)
Anirudh further illustrates the separate training phases in FIG. 3, which recites:
“train CMN” and “learn latent vectors.” (Anirudh, FIG. 3, steps 303 and 304)
Thus, Anirudh teaches a training process having separate phases for updating neural-network weight parameters and updating latent vectors.
It would have been obvious to apply Anirudh’s alternating parameter-update schedule to Hoshen’s joint optimization of generator weight parameters and per-image latent vectors. Hoshen teaches that both the generator weights
G
and latent vectors
z
i
are variables of the same optimization objective, and further recognizes alternating optimization as a known technique for jointly optimizing model parameters and latent representations. Anirudh teaches a predictable implementation in which network parameters are updated using gradient descent in one phase and latent vectors are updated using gradient descent in a separate phase.
A person of ordinary skill in the art would have had reason to alternately update Hoshen’s generator weight parameters and latent vectors to simplify optimization of the two respective parameter sets, permit each parameter set to be optimized relative to the current values of the other parameter set, and promote stable convergence of the joint training objective. Such modification would have involved applying a known alternating or block-coordinate optimization technique to Hoshen’s jointly trained generator weights and latent vectors, with a reasonable expectation of success.
Anirudh is relied upon for the alternating training schedule, rather than for the particular identity of Hoshen’s generator or the loss functions supplied by Hoshen and Kingma. The resulting method retains Hoshen’s generator, latent vectors, VGG perceptual loss, and encoder-less, non-adversarial training framework while alternately updating the generator weight parameters and latent vectors according to the known update schedule taught by Anirudh.
Regarding claims 7-12
Claims 7-11 are rejected under 35 U.S.C. §103 as being unpatentable over Hoshen in view of Gao and further in view of Kingma for the same reasons set forth above for corresponding method claims 1-5.
Claim 12 is rejected under 35 U.S.C. §103 as being unpatentable over Hoshen in view of Gao and Kingma, and further in view of Anirudh, for the same reasons set forth above for corresponding method claim 6.
Each of claims 7-12 is the system analog of a previously analyzed method claim:
claim 7 corresponds to claim 1;
claim 8 corresponds to claim 2;
claim 9 corresponds to claim 3;
claim 10 corresponds to claim 4;
claim 11 corresponds to claim 5; and
claim 12 corresponds to claim 6.
For each of claims 7-12, the recited one or more processors and memory storing instructions that, when executed by the one or more processors, cause the system to perform the claimed operations merely recite in system form the same functional limitations found obvious above for corresponding method claims 1-6.
Regarding claim 7, Hoshen in view of Gao and Kingma teaches the corresponding operations of claim 1, including associating each training image with a respective latent vector, generating an output image from the respective latent vector using a generator, iteratively updating generator weight parameters and latent vectors, calculating the recited generator and latent-space losses, and operating without a discriminator network, adversarial loss, or encoder network. Gao teaches modifying the generator to comprise residual connections and perform upsampling operations. Kingma teaches regularizing latent variables toward a preset target distribution. Implementing these known operations using one or more processors executing instructions stored in memory would have been a routine and predictable implementation of the computer-implemented training method taught by the combination.
Regarding claim 8, Hoshen in view of Gao and Kingma teaches the corresponding additional limitation of claim 2 for the same reasons set forth above. Hoshen teaches training latent vectors using an image-reconstruction loss and using normally distributed latent inputs, while Kingma teaches using reconstruction error and KL divergence to encourage latent variables to conform to a normal prior distribution.
Regarding claim 9, Hoshen in view of Gao and Kingma teaches the corresponding additional limitation of claim 3 for the same reasons set forth above. Hoshen teaches optimizing latent vectors to form a learned latent space, and Kingma teaches regularizing latent variables toward a normal prior distribution during training.
Regarding claim 10, Hoshen in view of Gao and Kingma teaches the corresponding additional limitation of claim 4 for the same reasons set forth above. Hoshen teaches a respective latent vector for each corresponding training image and randomly sampled normally distributed latent inputs, while Kingma teaches sampling latent variables from a Gaussian distribution having a defined scale or standard deviation.
Regarding claim 11, Hoshen in view of Gao and Kingma teaches the corresponding additional limitation of claim 5 for the same reasons set forth above. Hoshen teaches jointly training the generator and latent vectors based on a loss that establishes a mapping between latent space and image space, while Kingma teaches using KL-divergence regularization to encourage latent variables to conform to a preset prior distribution.
Regarding claim 12, Hoshen in view of Gao and Kingma, and further in view of Anirudh, teaches the corresponding additional limitation of claim 6 for the same reasons set forth above. Hoshen teaches training both generator weight parameters and latent vectors, while Anirudh teaches separately updating neural-network parameters and latent vectors in respective training phases. It would have been obvious to use Anirudh’s alternating update schedule when optimizing Hoshen’s generator weight parameters and latent vectors to simplify optimization of the respective parameter sets and promote stable convergence.
The recitation of one or more processors and memory does not patentably distinguish claims 7-12 from corresponding method claims 1-6. Hoshen expressly teaches computer-implemented neural-network training using stochastic gradient descent, generator weights, latent vectors, and image-generation operations. A person of ordinary skill in the art would have understood that such operations are performed by computing hardware executing stored program instructions. Reciting the same operations as instructions stored in memory and executed by one or more processors therefore represents a predictable system implementation of the methods found obvious above.
Accordingly, claims 7-11 are unpatentable over Hoshen in view of Gao and further in view of Kingma for the same reasons discussed above for claims 1-5, respectively, and claim 12 is unpatentable over Hoshen in view of Gao and Kingma, and further in view of Anirudh, for the same reasons discussed above for claim 6.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Paul Coleman whose telephone number is (571)272-4687. The examiner can normally be reached Mon-Fri.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL COLEMAN/ Examiner, Art Unit 2126
/DAVID YI/ Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights