DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s submission filed 2025-10-09 has been entered. The status of claims is as follows:
Claims 1-12 and 14-25 remain pending in the application.
Claims 1, 4, 16, and 23 are amended.
Claim 13 is cancelled.
Response to Arguments
Applicant’s arguments with respect to the rejections of Claims 1-12 and 14-15 under 35 USC 103 have been fully considered but they are not persuasive.
Applicant argues on Remarks Pages 10-12 that Xie does not teach the newly amended limitation “bounding the one or more other terms of the objective function based on the spectral regularization term using a product of a smoothness coefficient and a sum of first values representing largest predicted values of a plurality of layers included in the VAE”, as Xie is based on “spectral angle” and not a smoothness coefficient. Examiner agrees, but notes that this matter is derived from cancelled Claim 13, for which analogous matter was shown to be taught by Yoshida.
Applicant then argues that Yoshida does not teach this matter because Yoshida does not teach “bounding one or more other terms of an objective function based on a spectral regularization term.”
Examiner respectfully disagrees with this rebuttal based upon bounding one or more other terms, as discussed below.
Examiner has determined that this limitation should not be given patentable weight for the following reasons.
Specification [0040] defines the Loss function as:
PNG
media_image1.png
62
718
media_image1.png
Greyscale
Subsequently, Specification [0042] states: “the KL terms in objective function 220 can become unbounded and cause sharp gradient updates that destabilize training … to bound the KL terms, update component 212 may use spectral regularization 232 that minimizes the Lipschitz constant for each layer … spectral regularization 232 may be performed by adding the term
PNG
media_image2.png
28
126
media_image2.png
Greyscale
to Equation 1, where s(i) is the largest singular value of the ith conventional layer.”
Thus, Examiner notes that the claimed resulting loss function would be:
PNG
media_image3.png
48
675
media_image3.png
Greyscale
Examiner notes that KL divergence is an unbounded function, and in the equation above, the KL terms are not directly modified in any way by the SR term added at the end. As it stands, the loss function remains unbounded. Thus, a closer examination of the language is warranted. Spec [0042] states that the SR term is “to stabilize training”. Specification [0043] states: “selection of a suitable λ with spectral regularization 232 may reduce training instability caused by the KL terms in objective function 220 by ensuring that the output … does not change dramatically as the corresponding input changes.” Furthermore, Applicant’s paper “NVAE: A Deep Hierarchical Variational Autoencoder” by Vahdat and Kautz states on Page 5 Section 3.2 in the section “Spectral Regularization”: “We hypothesize that by regularizing the Lipschitz constant, we can ensure that the latent codes predicted by the encoder remain bounded, resulting in a stable KL minimization.”
Thus, it appears the intended result is that while KL is unbounded by its nature, the SR term added to the loss function may hypothetically prevent the KL terms from growing unbounded as training iterations progress. Examiner points out two passages from the MPEP. MPEP 2111.04 states: “whereby clause in a method claim is not given weight when it simply expresses the intended result of a process step positively recited”. This is applicable despite the absence of the word “whereby”. Furthermore, MPEP 2173.05(g) describes guidelines for functional limitations and states: “A claim term is functional when it recites a feature ‘by what it does rather than by what it is’ and “For example, when claims merely recite a description of a problem to be solved or a function or result achieved by the invention, the boundaries of the claim scope may be unclear” as well as “Examiners should consider the following factors when examining claims that contain functional language to determine whether the language is ambiguous … whether the language sets forth well-defined boundaries of the invention or only states a problem solved or a result obtained.”
To summarize, Examiner is not giving patentable weight to the limitation “that bounds one or more other terms of the objective function”, because it merely recites a hypothetical intended result of the positively recited step of “includes a spectral regularization term”.
As a result, Xie is being removed from the 103 rejections in this action, and the rejection is now based on a combination of only Kingma and Yoshida.
Applicant’s arguments with respect to the rejections of Claims 16-25 under 35 USC 103 have been fully considered but they are moot for two reasons.
One reason is that the amended matter requiring that “each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers” is rejected as new matter under 35 USC 112(a), as explained below.
The second reason is that the Luo reference, which has been argued against, has been replaced by Nachum, which teaches the former limitations better than Luo. This is because Luo teaches applying the regularization factor to the square of the scaling parameter gamma (γ2), but never applies the square root to it, which would be the L2 norm (L2 norm is the square root of the sum of squares of the vector elements). So Luo is very close to applying a norm to the scaling parameter, but not quite. Nachum, on the other hand, explicitly and directly applies a norm, the L1 norm (the sum of the absolute values of the vector elements), to the scaling factor gamma (|γj|1).
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 16-25 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 16 and 23 recite the limitation “the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers.” There is not sufficient support for this in the Specification. Examiner notes the following:
Spec [0046]: “update component 212 performs scaling regularization 234 that applies a regularization on a norm (ga, L1norm, L2norm, L-infinity norm, etc.) of scaling parameters in batch normalization layers of encoder 202”
Spec [0068]: “regularization of a scaling parameter associated with batch normalization of one or more layers of the VAE”
Spec [0069]: “In another example, training engine 122 may add a regularization term to the scaling parameter used to perform batch normalization in one or more layers of the encoder, prior, and/or decoder networks. The regularization term may include, but is not limited to, an L1norm, L2norm, and/or L-infinity norm. In turn, the regularization term may stabilize training of the VAE with respect to the batch normalization layer(s).”
The Specification in these areas merely states that “regularization” is applied via a “norm” of “scaling parameters” that are “used to perform batch normalization in one or more layers”. These parts of the Specification only state that a scaling parameter may be associated with one or more layers, and not that for regularization of the scaling factor, a norm is calculated on a plurality of scaling parameters, wherein each scaling parameter is associated with a different batch normalization layer.
This level of specificity does not exist, and these passages of the Specification appear to mean that regularization can be applied to a scaling factor, in which the scaling factor can apply to one or more layers, and a norm is calculated on the instances of a scaling parameter of a batch normalization layer.
Furthermore, since the scaling factor of a batch normalization layer is a vector, and a norm is a value calculated based on the elements of a vector (i.e., the square root of the sum of squared elements), it is unclear how one would calculate the norm of a plurality of vectors.
In summary, there is nothing in the Specification that indicates that a single regularization term is somehow applied to a plurality of batch normalization scaling factors each associated with a different layer.
Summary - 35 USC § 103
The claims are rejected under the following combinations of references (independent claims bold and underlined):
Claims
References
1, 4, 9
Kingma, Yoshida
2-3
Kingma, Yoshida, Hou
5, 8
Kingma, Yoshida, Chollet, Ioffe
6
Kingma, Yoshida, Chollet, Ioffe, Nachum
7, 11
Kingma, Yoshida, Chollet, Ioffe, Ramachandran
10, 12
Kingma, Yoshida, Chollet, Ioffe, Ramachandran, Hu
14
Kingma, Yoshida, Chen
15
Kingma, Yoshida, Micikevicius
16, 18-22
Kingma, Chollet, Ioffe, Nachum
17
Kingma, Chollet, Ioffe, Nachum, Yoshida
23-25
Kingma, Chollet, Ioffe, Ramachandran, Hu, Chen, Nachum
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Kingma et al. (“Improved variational inference with inverse autoregressive flow”; hereinafter “Kingma”) in view of Yoshida et al. (“Spectral norm regularization for improving the generalizability of deep learning”; hereinafter “Yoshida”)
As per Claim 1, Kingma teaches a method for performing machine learning, comprising:
inputting a set of training images into a machine learning model that comprises an encoder portion, a prior, and a decoder portion (Kingma, Page 7 Section 6.2, discloses the use of images: “We also evaluated IAF on the CIFAR-10 dataset of natural images.”
Kingma, top of Page 2, discloses training: “In particular, we train deep variational auto-encoders with latent variables at multiple levels of the hierarchy.”
Examiner notes that a “variational auto-encoder” comprises an encoder neural network and a decoder. Kingma also explicitly discloses these components at the bottom of Page 13: “The first layer of the encoder is a convolutional layer with 2 × 2 spatial subsampling; the last layer of the decoder has a matching convolutional layer with 2 × 2 spatial upsampling.” The encoder and decoder are neural networks as per Kingma, Page 1 Intro Para 1: “When using neural networks for both the inference network and generative model, this results in class of models called variational autoencoders (Kingma and Welling, 2013) (VAEs).” One of ordinary skill in the art will appreciate that the inference network is the encoder and the generative model is the decoder.
As for the prior, this is also a neural network. Note that Kingma on Page 7 Figure 3, as shown below, discloses a “Layer Prior”:
PNG
media_image4.png
236
400
media_image4.png
Greyscale
Furthermore, note that Kingma Page 11 Figure 4 gives further detail:
PNG
media_image5.png
338
325
media_image5.png
Greyscale
Note above that Kingma discloses “parameters of p(z|.)”, which is parameters of the prior distribution, which is then sampled from. Kingma, Top of Page 11, states: “Assuming L layers of latent variables, the generative model’s density function is factorized as p(x, z1, z2, z3, ...) = p(x, z1:L) = p(x|z1:L)p(z1:L). The second part of this density, the prior over the latent variable, is autoregressive: p(z1:L) = p(zL) ПL−1l=1 p(zl|zl+1:L). This autoregressive nature of the prior increases the flexibility of the true posterior, leading to improved empirical results. This improved performance is easily explained: as the true posterior is largely a function of this prior, a flexible prior improves the flexibility of the true posterior, making it easier for the VAE to match the approximate and true posteriors, leading to a tighter bound, without sacrificing the flexibility of the generative model itself.” Thus, Kingma discloses a “prior” with learned “parameters” that is “autoregressive”, and thus a neural network, as Kingma states in the Abstract: “The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network.” Furthermore, Kingma also discloses at the bottom of Page 7: “We sampled from the PixelCNN naïvely by sequentially generating a pixel at a time, using the full generative model at each iteration.” Thus, Kingma suggests sampling from the prior is achieved by sampling from a neural network (PixelCNN)).
Thus, Kingma discloses an encoder neural network, prior neural network, and decoder neural network.
training the machine learning model by updating one or more parameters of the machine learning model [wherein the objective function includes a spectral regularization term and one or more other terms, the objective function controlling a smoothness of one or more outputs produced by the machine learning model] when processing the set of training images, wherein training the machine learning model comprises: updating the one or more parameters of the machine learning model based on the objective function (Kingma, Page 7 Section 6.2, discloses the use of images: “We also evaluated IAF on the CIFAR-10 dataset of natural images.” Kingma, top of Page 2, discloses training: “In particular, we train deep variational auto-encoders with latent variables at multiple levels of the hierarchy.” Kingma, Top of Page 4 Algorithm 1, discloses: “neural network parameters”.)
producing a new image that reflects one or more visual attributes associated with the set of training images by applying the decoder portion to a value generated based on an output of the prior (Kingma, Page 11 Figure 4, discloses:
PNG
media_image5.png
338
325
media_image5.png
Greyscale
As shown above, Kingma discloses a layer that produces a new image (“Generative”) layer that is based on a visual attributes associated with the set of training images (“conv”, “slice features”), by applying the decoder portion (“Generative”) layer to a value generates based on an output of the prior (a “sample” from the prior p(z|.)”). Furthermore, Kingma also discloses at the bottom of Page 7: “We sampled from the PixelCNN naïvely by sequentially generating a pixel at a time, using the full generative model at each iteration.” Thus, Kingma suggests sampling from the prior is achieved by sampling from a neural network (PixelCNN).
Furthermore, Kingma discloses in Algorithm 1:
PNG
media_image6.png
32
279
media_image6.png
Greyscale
And thus discloses that the encoder produces a first distribution parameterized by
PNG
media_image7.png
20
44
media_image7.png
Greyscale
. Thus, the “prior” distribution disclosed above is a second distribution, that reflects the first distribution, and values are sampled from this second distribution (the prior), the prior being a distribution of latent variables (“Assuming L layers of latent variables”)).
However, Kingma does not teach wherein the objective function includes a spectral regularization term and one or more other terms, the objective function controlling a smoothness of one or more outputs produced by the machine learning model; bounding the one or more other terms of the objective function based on the spectral regularization term using a product of a smoothness coefficient and a sum of first values representing largest predicted values of a plurality of layers included in the machine learning model
Yoshida teaches wherein the objective function includes a spectral regularization term and one or more other terms, the objective function controlling a smoothness of one or more outputs produced by the machine learning model (Yoshida, Page 3 Section 3.2, discloses: “In this subsection, we explain spectral norm regularization. The notations are the same as those used in Section 3.1. To bound the spectral norm of each weight matrix, W`, we consider the following empirical risk minimization problem
PNG
media_image8.png
66
586
media_image8.png
Greyscale
where
PNG
media_image9.png
20
58
media_image9.png
Greyscale
is a regularization factor. We refer to the second term as the spectral norm regularizer. It decreases the spectral norms of the weight matrices. Here, Yoshida’s Eq 1 discloses an objective function that includes spectral regularization term (“second term as the spectral norm regularizer”).
Yoshida further discloses in Page 2 Section 3: “In this section, we explain spectral norm regularization and how it reduces the sensitivity to test data perturbation.” Here, Yoshida discloses that the output data is less “sensitive” to “perturbation” in input data, and therefore the outputs are smoother. Yoshida, at the top of Page 5, describes this as “smoothness” when discussing competing approaches to mitigating perturbation sensitivity: “smoothness of a model against input perturbation.”)
bounding the one or more other terms of the objective function based on the spectral regularization term using a product of a smoothness coefficient and a sum of first values representing largest predicted values of a plurality of layers included in the machine learning model (Examiner notes above that “bounding one or more other terms of the objective function” is not given patentable weight, as the claimed limitation results in a theoretical bounding of the entire KL divergence. Yoshida teaches the spectral regularization term using a product of a smoothness coefficient and a sum of first values representing largest predicted values of a plurality of layers included in the machine learning model as shown below.
Recall above that Kingma teaches a plurality of layers in a VAE. Yoshida, Page 3 Section 3.2, discloses: “To bound the spectral norm of each weight matrix, Wl, we consider the following empirical risk minimization problem:
PNG
media_image10.png
224
572
media_image10.png
Greyscale
where λ e R+ is a regularization factor. We refer to the second term as the spectral norm regularizer. It decreases the spectral norms of the weight matrices.” Regarding the “spectral norms”, Yoshida discloses on “The spectral norm of a matrix … is defined as … the largest singular value of A.” Therefore, when applying this technique to each of the plurality of layers established above by Kingma, the spectral regularization term includes a coefficient multiplied by a summation of a plurality of largest predicted values of a plurality of layers included in the machine learning model.)
Yoshida is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder of Kingma with the spectral norm regularization of Yoshida. One of ordinary skill in the art would be motivated to do so in order to achieve better generalizability of results, by reducing sensitivity to perturbations (Yoshida, Page 8 Section 5 Conclusion: “In this work, we hypothesized that a high sensitivity to the perturbation of the input data degrades the performance of the data. In order to reduce the sensitivity to the perturbation of the test data, we proposed the spectral norm regularization method, and confirmed that it exhibits a better generalizability than other baseline methods through experiments. Experimental comparison with other methods indicated that the insensitivity to the perturbation of the test data plays a crucial role in determining the generalizability.”)
As per Claim 4, Kingma teaches a method for performing machine learning, comprising:
inputting a training dataset into a variational autoencoder (VAE) comprising an encoder network, a prior network, and a decoder network (Kingma, Page 7 Section 6.2, discloses the use of images: “We also evaluated IAF on the CIFAR-10 dataset of natural images.”
Kingma, top of Page 2, discloses training: “In particular, we train deep variational auto-encoders with latent variables at multiple levels of the hierarchy.”
Examiner notes that a “variational auto-encoder” comprises an encoder neural network and a decoder. Kingma also explicitly discloses these components at the bottom of Page 13: “The first layer of the encoder is a convolutional layer with 2 × 2 spatial subsampling; the last layer of the decoder has a matching convolutional layer with 2 × 2 spatial upsampling.” The encoder and decoder are neural networks as per Kingma, Page 1 Intro Para 1: “When using neural networks for both the inference network and generative model, this results in class of models called variational autoencoders (Kingma and Welling, 2013) (VAEs).” One of ordinary skill in the art will appreciate that the inference network is the encoder and the generative model is the decoder.
As for the prior, this is also a neural network. Note that Kingma on Page 7 Figure 3, as shown below, discloses a “Layer Prior”:
PNG
media_image4.png
236
400
media_image4.png
Greyscale
Furthermore, note that Kingma Page 11 Figure 4 gives further detail:
PNG
media_image5.png
338
325
media_image5.png
Greyscale
Note above that Kingma discloses “parameters of p(z|.)”, which is parameters of the prior distribution, which is then sampled from. Kingma, Top of Page 11, states: “Assuming L layers of latent variables, the generative model’s density function is factorized as p(x, z1, z2, z3, ...) = p(x, z1:L) = p(x|z1:L)p(z1:L). The second part of this density, the prior over the latent variable, is autoregressive: p(z1:L) = p(zL) ПL−1l=1 p(zl|zl+1:L). This autoregressive nature of the prior increases the flexibility of the true posterior, leading to improved empirical results. This improved performance is easily explained: as the true posterior is largely a function of this prior, a flexible prior improves the flexibility of the true posterior, making it easier for the VAE to match the approximate and true posteriors, leading to a tighter bound, without sacrificing the flexibility of the generative model itself.” Thus, Kingma discloses a “prior” with learned “parameters” that is “autoregressive”, and thus a neural network, as Kingma states in the Abstract: “The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network.” Furthermore, Kingma also discloses at the bottom of Page 7: “We sampled from the PixelCNN naïvely by sequentially generating a pixel at a time, using the full generative model at each iteration.” Thus, Kingma suggests sampling from the prior is achieved by sampling from a neural network (PixelCNN)).
Thus, Kingma discloses an encoder neural network, prior neural network, and decoder neural network.
training the VAE by updating one or more parameters of the VAE based on an objective function, [wherein the objective function includes a spectral regularization term and one or more other terms, the objective function controlling a smoothness of one or more outputs produced by the VAE when processing the training dataset], wherein the training the VAE comprises updating the one or more parameters of the VAE based on the objective function (Kingma, Page 7 Section 6.2, discloses the use of images: “We also evaluated IAF on the CIFAR-10 dataset of natural images.” Kingma, top of Page 2, discloses training: “In particular, we train deep variational auto-encoders with latent variables at multiple levels of the hierarchy.” Kingma, Top of Page 4 Algorithm 1, discloses: “neural network parameters”.)
producing generative output that reflects a first distribution of the training dataset by applying the decoder network to one or more values sampled from a second distribution of latent variables generated by the prior network (Kingma, Page 11 Figure 4, discloses:
PNG
media_image5.png
338
325
media_image5.png
Greyscale
As shown above, Kingma discloses a layer that produces a new image (“Generative”) layer that is based on a visual attributes associated with the set of training images (“conv”, “slice features”), by applying the decoder portion (“Generative”) layer to a value generates based on an output of the prior (a “sample” from the prior p(z|.)”). Furthermore, Kingma also discloses at the bottom of Page 7: “We sampled from the PixelCNN naïvely by sequentially generating a pixel at a time, using the full generative model at each iteration.” Thus, Kingma suggests sampling from the prior is achieved by sampling from a neural network (PixelCNN).
Furthermore, Kingma discloses in Algorithm 1:
PNG
media_image6.png
32
279
media_image6.png
Greyscale
And thus discloses that the encoder produces a first distribution parameterized by
PNG
media_image7.png
20
44
media_image7.png
Greyscale
. Thus, the “prior” distribution disclosed above is a second distribution, that reflects the first distribution, and values are sampled from this second distribution (the prior), the prior being a distribution of latent variables (“Assuming L layers of latent variables”)).
However, Kingma does not teach wherein the objective function includes a spectral regularization term and one or more other terms, the objective function controlling a smoothness of one or more outputs produced by the VAE when processing the training dataset; bounding the one or more other terms of the objective function based on the spectral regularization term using a product of a smoothness coefficient and a sum of first values representing largest predicted values of a plurality of layers included in the machine learning model
Yoshida teaches wherein the objective function includes a spectral regularization term and one or more other terms, the objective function controlling a smoothness of one or more outputs produced by the VAE when processing the training dataset Recall above that Kingma teaches a VAE. Yoshida, Page 3 Section 3.2, discloses: “In this subsection, we explain spectral norm regularization. The notations are the same as those used in Section 3.1. To bound the spectral norm of each weight matrix, W`, we consider the following empirical risk minimization problem
PNG
media_image8.png
66
586
media_image8.png
Greyscale
where
PNG
media_image9.png
20
58
media_image9.png
Greyscale
is a regularization factor. We refer to the second term as the spectral norm regularizer. It decreases the spectral norms of the weight matrices. Here, Yoshida’s Eq 1 discloses an objective function that includes spectral regularization term (“second term as the spectral norm regularizer”).
Yoshida further discloses in Page 2 Section 3: “In this section, we explain spectral norm regularization and how it reduces the sensitivity to test data perturbation.” Here, Yoshida discloses that the output data is less “sensitive” to “perturbation” in input data, and therefore the outputs are smoother. Yoshida, at the top of Page 5, describes this as “smoothness” when discussing competing approaches to mitigating perturbation sensitivity: “smoothness of a model against input perturbation.”)
bounding the one or more other terms of the objective function based on the spectral regularization term using a product of a smoothness coefficient and a sum of first values representing largest predicted values of a plurality of layers included in the machine learning model (Examiner notes above that “bounding one or more other terms of the objective function” is not given patentable weight, as the claimed limitation results in a theoretical bounding of the entire KL divergence. Yoshida teaches the spectral regularization term using a product of a smoothness coefficient and a sum of first values representing largest predicted values of a plurality of layers included in the machine learning model as shown below.
Recall above that Kingma teaches a plurality of layers in a VAE. Yoshida, Page 3 Section 3.2, discloses: “To bound the spectral norm of each weight matrix, Wl, we consider the following empirical risk minimization problem:
PNG
media_image10.png
224
572
media_image10.png
Greyscale
where λ e R+ is a regularization factor. We refer to the second term as the spectral norm regularizer. It decreases the spectral norms of the weight matrices.” Regarding the “spectral norms”, Yoshida discloses on “The spectral norm of a matrix … is defined as … the largest singular value of A.” Therefore, when applying this technique to each of the plurality of layers established above by Kingma, the spectral regularization term includes a coefficient multiplied by a summation of a plurality of largest predicted values of a plurality of layers included in the machine learning model.)
Yoshida is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder of Kingma with the spectral norm regularization of Yoshida. One of ordinary skill in the art would be motivated to do so in order to achieve better generalizability of results, by reducing sensitivity to perturbations (Yoshida, Page 8 Section 5 Conclusion: “In this work, we hypothesized that a high sensitivity to the perturbation of the input data degrades the performance of the data. In order to reduce the sensitivity to the perturbation of the test data, we proposed the spectral norm regularization method, and confirmed that it exhibits a better generalizability than other baseline methods through experiments. Experimental comparison with other methods indicated that the insensitivity to the perturbation of the test data plays a crucial role in determining the generalizability.”)
As per Claim 9, the combination of Kingma and Yoshida the method of claim 4. Kingma teaches wherein the VAE comprises a hierarchy of groups of the latent variables, and wherein a first sample from a first group in the hierarchy is combined with a feature map and passed to a second group following the first group in the hierarchy for use in generating a second sample from the second group. (Kingma, Page 5 Figure 2, discloses:
PNG
media_image11.png
228
931
media_image11.png
Greyscale
As shown above, Kingma discloses a hierarchy of groups of latent variables (µ, σ, h), where a first sample (z) is combined with a feature map (h) and passed to a second group (“IAF Step”) for use in generating a second sample (z)).
Claims 2-3 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma and Yoshida, further in view of Hou et al. (“Deep Feature Consistent Variational Autoencoder”; hereinafter “Hou”)
As per Claim 2, the combination of Kingma and Yoshida teaches the method of claim 1. However, the combination does not teach wherein the new image comprises a face that is not found in the set of training images.
Hou teaches wherein the new image comprises a face that is not found in the set of training images. (Hou, Page 1140, Conclusion, discloses: “In this paper, we propose to train a deep feature consistent variational autoencoder by feature perceptual loss based on pretrained deep CNNs to measure the similarity of the input and generated images. We apply our model on face image generation.”)
Hou is analogous art because it is in the field of endeavor of machine learning and variational autoencoders. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual blocks and spectral norm regularization of Kingma and Yoshida with the application to facial generation of Hou, who also uses a VAE. One of ordinary skill in the art would be motivated to do so in order to take advantage of the benefits of the VAE architecture of Kingma and Yoshida by applying it to generate realistic faces (Hou, Page 1140 Conclusion: “We apply our model on face image generation and achieve comparable and even better performance when compared to different generative models. In addition, we explore the learned latent representation in our model and demonstrate that it has powerful capability of capturing the conceptual and semantic information of natural (face) images. We also achieve state-of-the-art performance in facial attribute prediction based on the learned latent representation.”)
As per Claim 3, the combination of Kingma and Yoshida teaches the method of claim 1. However, the combination does not teach wherein the new image comprises an animal or a vehicle that is not found in the set of training images.
Hou teaches wherein the new image comprises an animal or a vehicle that is not found in the set of training images. (Hou, Page 1140, Conclusion, discloses: “In this paper, we propose to train a deep feature consistent variational autoencoder by feature perceptual loss based on pretrained deep CNNs to measure the similarity of the input and generated images. We apply our model on face image generation.” Hou generates human faces, and humans are animals.)
Hou is analogous art because it is in the field of endeavor of machine learning and variational autoencoders. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual blocks and spectral norm regularization of Kingma and Yoshida with the application to facial generation of Hou, who also uses a VAE. One of ordinary skill in the art would be motivated to do so in order to take advantage of the benefits of the VAE architecture of Kingma and Yoshida by applying it to generate realistic faces (Hou, Page 1140 Conclusion: “We apply our model on face image generation and achieve comparable and even better performance when compared to different generative models. In addition, we explore the learned latent representation in our model and demonstrate that it has powerful capability of capturing the conceptual and semantic information of natural (face) images. We also achieve state-of-the-art performance in facial attribute prediction based on the learned latent representation.”)
Claims 5 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma and Yoshida, further in view of Chollet et al. (“Xception: Deep Learning With Depthwise Separable Convolutions”; hereinafter “Chollet”), and further in view of Ioffe (“Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models”).
As per Claim 5, the combination of Kingma and Yoshida teaches the method of claim 4. Kingma teaches wherein applying the decoder network to the one or more values comprises applying [batch] normalization to one or more layers of the decoder network [based on a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization] (As shown in the rejection to claim 4, Kingma teaches that a decoder network (“generative layer”) generates parameters of a second distribution of data (“prior”) based on the values sampled from the first distribution. Kingma also discloses to apply normalization, but not batch normalization (“We also found that the noise introduced by batch normalization hurts performance; instead we use weight normalization (Salimans and Kingma, 2016) method”).
Examiner notes that another reference (Chollet) will teach batch normalization. And another reference (Ioffe) will be applied that will describe how to alleviate the “hurt performance” for batch normalization.)
However, Kingma does not teach applying batch normalization … based on a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization
Chollet teaches applying batch normalization to one or more layers of the decoder network (Recall above that Kingma discloses residual units comprising a decoder network comprising convolutional layers. Chollet, Page 1255 Figure 5 caption, discloses: “Note that all Convolution and SeparableConvolution layers are followed by batch normalization [7] (not included in the diagram)”.
Chollet is analogous art because it is in the field of endeavor of machine learning and deep residual networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual blocks and spectral norm regularization of Kingma and Yoshida with the batch normalization of Chollet. Chollet teaches to perform batch normalization after every convolutional layer, and in combination with Kingma’s decoder network which comprises convolutional networks, this would result in applying batch normalization to layers of the decoder network. One of ordinary skill in the art will appreciate that batch normalization is very well known in the art, and one would be motivated to do so in order to improve training performance, as indicated by one of the inventors of batch normalization, Ioffe, who is also cited below for momentum parameter (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models.”) As for Kingma’s stated difficulties regarding it, this problem of the combination will be solved by further combination with Ioffe shown below.
However, the combination of Kingma, Yoshida, and Chollet does not teach applying batch normalization … based on a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization
Ioffe teaches applying batch normalization … based on a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”.)
Ioffe is analogous art because it is in the field of endeavor of machine learning and batch normalization (“Batch Renormalization”). It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual blocks, spectral norm regularization, and batch normalization of Kingma, Yoshida, and Chollet with the batch renormalization with “moving average update rate α” of Ioffe, and applying it after all convolutions, including those of the decoder. Kingma notes that “we also found that the noise introduced by batch normalization hurts performance”. Ioffe’s “moving average update rate α” alleviates this problem of “noise” by smoothing out the average into a moving average. One of ordinary skill in the art would be motivated to adjust Ioffe’s update rate in order to gain the advantages of batch normalization in all parts of the VAE including the decoder (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models”), while avoiding the problems of noise in the minibatches (Ioffe, Abstract: “Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches.”) Thus, one of ordinary skill in the art would understand that adjusting this parameter would alleviate Kingma’s concerns about batch normalization, rendering the combination obvious.
As per Claim 8, the combination of Kingma, Yoshida, Chollet, and Ioffe teaches the method of claim 5 as well as batch normalization based on the one or more values sampled from the second distribution (see rejection to claim 5). Ioffe teaches wherein applying the batch normalization to the one or more layers of the decoder network comprises recalculating batch statistics associated with the batch normalization based on the one or more values sampled from the second distribution. (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”.)
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma, Yoshida, Chollet, and Ioffe, further in view of Nachum et al. (US 2019/0147339 A1; hereinafter “Nachum”).
As per Claim 6, the combination of Kingma, Yoshida, Chollet, and Ioffe teaches the method of claim 5 as well as training the VAE (see rejection to claim 4). However, the combination does not teach wherein training the VAE comprises applying a regularization parameter to a scaling parameter associated with the batch normalization.
Nachum teaches wherein training the VAE comprises applying a regularization parameter to a scaling parameter associated with the batch normalization (Nachum [0007]: “In some implementations, the terms of the shrinking engine loss function that penalize active neurons of the neural network comprise: a batch normalization regularization term comprising a sparsity-inducing norm of a scale parameter of a batch normalization layer of the neural network.”)
Nachum is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Kingma, Yoshida, Chollet, and Ioffe with Nachum. One of ordinary skill in the art would have been motivated to perform regularization on the batch normalization scaling parameter as known in the art via previous work by Luo and van Laarhoven, as it serves to improve the learning rate (Luo Page 9 Conclusion: “This work investigated an explicit regularization form of BN, which was decomposed into PN and gamma decay where the regularization strengths from B and B were explored. Moreover, optimization and generalization of BN with regularization were derived and compared with vanilla SGD, WN, and WN+gamma decay, showing that BN enables training to converge with large maximum and effective learning rate, as well as leads to better generalization.”) Furthermore, Luo cites Van Laarhoven (“Towards Understanding Regularization in Batch Normalization”) on Page 2 (“Moreover, van Laarhoven (2017) showed that weight decay has no regularization effect when using together with BN or WN”) and Page 5 (“The effective LRs shown in Table 1 are consistent with previous work (van Laarhoven, 2017.))” Examiner notes that Van Laarhoven already suggested regularizing the scaling parameter (Van Laarhoven, Top of Page 2, “In this paper we investigate the effects of L2 regularization in combination with Batch, Weight and Layer Normalization. We show that, as expected, there is no regularizing effect. Rather, the ‘regularization’ term strongly influences the learning rate” and Page 3 Above Section 4, “We can now also answer the question of why L2 regularization is still beneficial when training neural networks with Batch Normalization: If no regularization is used the weights can grow unbounded, and the effective learning rate goes to 0.”) Furthermore, Van Laarhoven, Bottom of Page 8 states: “With batch normalization we have added two additional parameters, γ and β, and it of course makes sense to also regularize these. In our experiments we did not use regularization for these parameters, though preliminary experiments show that regularization here does not affect the results.”)
Claims 7 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma, Yoshida, Chollet, and Ioffe, further in view of Ramachandran et al. (“Searching for Activation Functions”; hereinafter “Ramachandran”). Examiner note: In Claim 11 both Chollet and Ioffe are recited to teach batch normalization. Kingma teaches residual blocks, and Chollet teaches that batch normalization be used after each convolution in a residual block. Meanwhile Ioffe teaches a particular variant of batch normalization that overcomes Kingma’s stated difficulties with it. Including Ioffe also offers consistency with the combination of references used for rejections in claims 5-8, 18, and 23-25, which explicitly recite Ioffe’s “momentum parameter”. Details are provided below.
As per Claim 7, the combination of Kingma, Yoshida, Chollet, and Ioffe teaches the method of claim 5. Chollet teaches wherein applying the batch normalization to the one or more layers comprises combining the batch normalization with a [Swish] activation function. (Chollet, Page 1255 Figure 5 Caption, discloses: “Note that all Convolution and SeparableConvolution layers are followed by batch normalization [7] (not included in the diagram)”. Furthermore, Chollet in the same Figure 5, discloses “ReLU” layers after each convolution, which are activation functions. Therefore, Chollet discloses batch normalization combined with an activation function.)
However, Chollet does not teach Swish activation function.
Ramachandran teaches Swish activation function (Ramachandran, Top of Page 2, discloses: “The best discovered activation function, which we call Swish, is f(x) = x sigmoid(x), where is a constant or trainable parameter.”)
Ramachandran is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, spectral norm regularization, and ReLU activation of Kingma, Yoshida, Chollet, and Ioffe with the Swish activation of Ramachandran, by simply replacing the ReLU used by Chollet with the Swish of Ramachandran. One of ordinary skill in the art would be motivated to do so in order to gain improved accuracy (Ramachandran, Top of Page 2: “Our extensive experiments show that Swish consistently matches or outperforms ReLU on deep networks applied to a variety of challenging domains such as image classification and machine translation. On ImageNet, replacing ReLUs with Swish units improves top-1 classification accuracy by 0.9% on Mobile NASNet-A (Zoph et al., 2017) and 0.6% on Inception-ResNet-v2 (Szegedy et al., 2017). These accuracy gains are significant given that one year of architectural tuning and enlarging yielded 1.3% accuracy improvement going from Inception V3 (Szegedy et al., 2016) to Inception-ResNet-v2 (Szegedy et al., 2017).”)
As per Claim 11, the combination of Kingma and Yoshida teaches the method of claim 4. Kingma teaches wherein the VAE comprises a residual cell (Kingma, Bottom of Page 10, discloses: “For CIFAR-10, we used a novel neural variational autoencoder (VAE) architecture with ResNet (He et al., 2015, 2016) units and multiple stochastic layers. Our architecture consists of L stacked blocks, where each block (l = 1..L) is a combination of a bottom-up residual unit for inference, producing a series of bottom-up activations h(q) l , and a top-down residual unit used for both inference and generation, producing a series of top-down activations h(p) l . The hidden layer of each residual function in the generative model contains a combination of the usual deterministic hidden units and a relatively small number of stochastic hidden units with a heteroscedastic diagonal Gaussian distribution p(zl|h(p) l ) given the unit’s input h(p) l , followed by a nonlinearity. We utilize wide (Zagoruyko and Komodakis, 2016) pre-activation residual units (He et al., 2015) with single-hidden-layer residual functions.”
Here, Kingma discloses a first residual cell (“residual unit”, “hidden layer of each residual function in the generative model contains a combination of the usual deterministic hidden units”, “pre-activation residual units”)).
However, Kingma does not teach the residual cell comprises a first BN layer, a first convolutional layer following the first BN layer, a second BN layer with a first Swish activation function, and a depthwise separable convolution layer following the second BN layer.
Chollet teaches the residual cell comprises a first BN layer, a first convolutional layer following the first BN layer, a second BN layer with a first [Swish] activation function (Chollet, Page 1255 Figure 5, discloses:
PNG
media_image12.png
642
1146
media_image12.png
Greyscale
Chollet, as shown above, discloses that “Note that all Convolution and SeparableConvolution layers are followed by batch normalization [7] (not included in the diagram)”, and thus Examiner has indicated where they would appear in the diagram (after convolutions, before ReLU). Furthermore, Chollet, discloses the use of residual cells in Page 1253 Section 3 Para 2: “The 36 convolutional layers are structured into 14 modules, all of which have linear residual connections around them, except for the first and last modules.”)
depthwise separable convolution layer following the second BN layer (Regarding the convolution layers, Chollet discloses on Page 1253 Section 3: “We propose a convolutional neural network architecture based entirely on depthwise separable convolution layers.” Furthermore, on Figure 5, Chollet labels these layers “SeparableConv”.)
Chollet is analogous art because it is in the field of endeavor of machine learning and deep convolutional neural networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder of Kingma with the depthwise separable convolutions and batch normalization of Chollet, by incorporating Chollet’s architecture of residual blocks into the residual blocks of Kingma. One of ordinary skill in the art would have been motivated to do so in order to gain increased performance at a low cost of effort (Chollet, Page 1257, Conclusion: “Compared to Inception V3, Xception shows small gains in classification performance on the ImageNet dataset and large gains on the JFT dataset. We expect depthwise separable convolutions to become a cornerstone of convolutional neural network architecture design in the future, since they offer similar properties as Inception modules, yet are as easy to use as regular convolution layers.”) Examiner further points out that depthwise separable convolutions have become well known in the art for their efficiency, as for example, another reference Sandler et al. (“MobileNetV2: Inverted Residuals and Linear Bottlenecks”) discloses in Page 4511 Section 3.1: “Depthwise Separable Convolutions are a key building block for many efficient neural network architectures [27, 28, 20] and we use them in the present work as well … Effectively depthwise separable convolution reduces computation compared to traditional layers by almost a factor of k2.” Furthermore, Chollet teaches to perform batch normalization after every convolutional layer, and in combination with Kingma’s decoder network which comprises convolutional networks, this would result in applying batch normalization to layers of the decoder network. One of ordinary skill in the art will appreciate that batch normalization is very well known in the art, and one would be motivated to do so in order to improve training performance, as indicated by one of the inventors of batch normalization, Ioffe, who is also cited below for momentum parameter (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models.”) As for Kingma’s stated difficulties regarding it, this problem of the combination will be solved by further combination with Ioffe shown below.
Ioffe also teaches batch normalization (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”.)
Ioffe is analogous art because it is in the field of endeavor of machine learning and batch normalization (“Batch Renormalization”). It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions and batch normalization of Kingma, Yoshida, and Chollet with the batch renormalization with “moving average update rate α” of Ioffe. While Chollet discloses “note that all Convolution and SeparableConvolution layers are followed by batch normalization”, Kingma notes that “we also found that the noise introduced by batch normalization hurts performance”. Ioffe’s “moving average update rate α” alleviates this problem of “noise” by smoothing out the average into a moving average. One of ordinary skill in the art would be motivated to adjust Ioffe’s update rate in order to gain the advantages of batch normalization (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models”), while avoiding the problems of noise in the minibatches (Ioffe, Abstract: “Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches.”) Thus, one of ordinary skill in the art would understand that adjusting this parameter would alleviate Kingma’s concerns about batch normalization, rendering the combination also with Chollet (who also teaches batch normalization), obvious.
However, the combination of Kingma, Yoshida, Chollet, and Ioffe does not teach Swish activation function
Ramachandran teaches Swish activation function (Ramachandran, Top of Page 2, discloses: “The best discovered activation function, which we call Swish, is f(x) = x sigmoid(x), where is a constant or trainable parameter.”)
Ramachandran is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, and ReLU activation of Kingma and Chollet with the Swish activation of Ramachandran, by simply replacing the ReLU used by Chollet with the Swish of Ramachandran. One of ordinary skill in the art would be motivated to do so in order to gain improved accuracy (Ramachandran, Top of Page 2: “Our extensive experiments show that Swish consistently matches or outperforms ReLU on deep networks applied to a variety of challenging domains such as image classification and machine translation. On ImageNet, replacing ReLUs with Swish units improves top-1 classification accuracy by 0.9% on Mobile NASNet-A (Zoph et al., 2017) and 0.6% on Inception-ResNet-v2 (Szegedy et al., 2017). These accuracy gains are significant given that one year of architectural tuning and enlarging yielded 1.3% accuracy improvement going from Inception V3 (Szegedy et al., 2016) to Inception-ResNet-v2 (Szegedy et al., 2017).”)
Claim 10 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma and Yoshida, further in view of Chollet, further in view of Ioffe (“Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models”), further in view of Ramachandran and further in view of Hu et al. (“Squeeze-and-Excitation Networks”; hereinafter “Hu”). Examiner note: In these rejections both Chollet and Ioffe are recited to teach batch normalization. Kingma teaches residual blocks, and Chollet teaches that batch normalization be used after each convolution in a residual block. Meanwhile Ioffe teaches a particular variant of batch normalization that overcomes Kingma’s stated difficulties with it. Including Ioffe also offers consistency with the combination of references used for rejections in claims 5-8, 18, and 23-25, which explicitly recite Ioffe’s “momentum parameter”. Details are provided below.
As per Claim 10, the combination of Kingma and Yoshida teaches the method of claim 4. Kingma teaches wherein the VAE comprises a residual cell (Kingma, Bottom of Page 10, discloses: “For CIFAR-10, we used a novel neural variational autoencoder (VAE) architecture with ResNet (He et al., 2015, 2016) units and multiple stochastic layers. Our architecture consists of L stacked blocks, where each block (l = 1..L) is a combination of a bottom-up residual unit for inference, producing a series of bottom-up activations h(q) l , and a top-down residual unit used for both inference and generation, producing a series of top-down activations h(p) l . The hidden layer of each residual function in the generative model contains a combination of the usual deterministic hidden units and a relatively small number of stochastic hidden units with a heteroscedastic diagonal Gaussian distribution p(zl|h(p) l ) given the unit’s input h(p) l , followed by a nonlinearity. We utilize wide (Zagoruyko and Komodakis, 2016) pre-activation residual units (He et al., 2015) with single-hidden-layer residual functions.”
Here, Kingma discloses a first residual cell (“residual unit”, “hidden layer of each residual function in the generative model contains a combination of the usual deterministic hidden units”, “pre-activation residual units”)).
However, Kingma does not teach and the residual cell comprises a first batch-normalization (BN) layer with a first Swish activation function, a first convolutional layer following the first BN layer with the first Swish activation function, a second BN layer with a second Swish activation function, a second convolutional layer following the second BN layer with the second Swish activation function, and a squeeze and excitation (SE) layer.
Chollet teaches and the first residual cell comprises a first batch normalization (BN) layer with a first [Swish] activation function, a first convolutional layer following the first BN layer with the first [Swish] activation function, a second BN layer with a second [Swish] activation function, a second convolutional layer following the second BN layer with the second [Swish] activation function (Chollet, Page 1255 Figure 5, discloses:
PNG
media_image12.png
642
1146
media_image12.png
Greyscale
Chollet, as shown above, discloses that “Note that all Convolution and SeparableConvolution layers are followed by batch normalization [7] (not included in the diagram)”, and thus Examiner has indicated where they would appear in the diagram (after convolutions, before ReLU). Furthermore, Chollet, discloses the use of residual cells in Page 1253 Section 3 Para 2: “The 36 convolutional layers are structured into 14 modules, all of which have linear residual connections around them, except for the first and last modules.”)
Chollet is analogous art because it is in the field of endeavor of machine learning and deep convolutional neural networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder of Kingma with the depthwise separable convolutions and batch normalization of Chollet, by incorporating Chollet’s architecture of residual blocks into the residual blocks of Kingma. One of ordinary skill in the art would have been motivated to do so in order to gain increased performance at a low cost of effort (Chollet, Page 1257, Conclusion: “Compared to Inception V3, Xception shows small gains in classification performance on the ImageNet dataset and large gains on the JFT dataset. We expect depthwise separable convolutions to become a cornerstone of convolutional neural network architecture design in the future, since they offer similar properties as Inception modules, yet are as easy to use as regular convolution layers.”) Examiner further points out that depthwise separable convolutions have become well known in the art for their efficiency, as for example, another reference Sandler et al. (“MobileNetV2: Inverted Residuals and Linear Bottlenecks”) discloses in Page 4511 Section 3.1: “Depthwise Separable Convolutions are a key building block for many efficient neural network architectures [27, 28, 20] and we use them in the present work as well … Effectively depthwise separable convolution reduces computation compared to traditional layers by almost a factor of k2.” Furthermore, Chollet teaches to perform batch normalization after every convolutional layer, and in combination with Kingma’s decoder network which comprises convolutional networks, this would result in applying batch normalization to layers of the decoder network. One of ordinary skill in the art will appreciate that batch normalization is very well known in the art, and one would be motivated to do so in order to improve training performance, as indicated by one of the inventors of batch normalization, Ioffe, who is also cited below for momentum parameter (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models.”) As for Kingma’s stated difficulties regarding it, this problem of the combination will be solved by further combination with Ioffe shown below.
Ioffe also teaches batch normalization (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”.)
Ioffe is analogous art because it is in the field of endeavor of machine learning and batch normalization (“Batch Renormalization”). It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions and batch normalization of Kingma, Yoshida, and Chollet with the batch renormalization with “moving average update rate α” of Ioffe. While Chollet discloses “note that all Convolution and SeparableConvolution layers are followed by batch normalization”, Kingma notes that “we also found that the noise introduced by batch normalization hurts performance”. Ioffe’s “moving average update rate α” alleviates this problem of “noise” by smoothing out the average into a moving average. One of ordinary skill in the art would be motivated to adjust Ioffe’s update rate in order to gain the advantages of batch normalization (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models”), while avoiding the problems of noise in the minibatches (Ioffe, Abstract: “Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches.”) Thus, one of ordinary skill in the art would understand that adjusting this parameter would alleviate Kingma’s concerns about batch normalization, rendering the combination also with Chollet (who also teaches batch normalization), obvious.
However, the combination of Kingma, Yoshida, Chollet, and Ioffe does not teach Swish activation function; a squeeze and excitation (SE) layer
Ramachandran teaches Swish activation function (Ramachandran, Top of Page 2, discloses: “The best discovered activation function, which we call Swish, is f(x) = x sigmoid(x), where is a constant or trainable parameter.”)
Ramachandran is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, and ReLU activation of Kingma and Chollet with the Swish activation of Ramachandran, by simply replacing the ReLU used by Chollet with the Swish of Ramachandran. One of ordinary skill in the art would be motivated to do so in order to gain improved accuracy (Ramachandran, Top of Page 2: “Our extensive experiments show that Swish consistently matches or outperforms ReLU on deep networks applied to a variety of challenging domains such as image classification and machine translation. On ImageNet, replacing ReLUs with Swish units improves top-1 classification accuracy by 0.9% on Mobile NASNet-A (Zoph et al., 2017) and 0.6% on Inception-ResNet-v2 (Szegedy et al., 2017). These accuracy gains are significant given that one year of architectural tuning and enlarging yielded 1.3% accuracy improvement going from Inception V3 (Szegedy et al., 2016) to Inception-ResNet-v2 (Szegedy et al., 2017).”)
However, the combination of Kingma, Yoshida, Chollet, Ioffe, and Ramachandran does not teach a squeeze and excitation (SE) layer
Hu teaches a squeeze and excitation (SE) layer (Hu, Page 9 Right Column, Para 2, discloses: “We also construct a variant of the design which moves the SE block inside the residual unit, placing it directly after the 3 x 3 convolutional layer.”)
Hu is analogous art because it is in the field of endeavor of machine learning and convolutional neural networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, and Swish activation of Kingma, Chollet, and Ramachandran with the squeeze and excitation layer of Hu to be placed after the convolutional layer of the residual unit. One of ordinary skill in the art would be motivated to do so in order to improve the quality of results by focusing on the most informative features (Hu, Page 1 Right Column Para 1: “In this paper, we investigate a different aspect of network design - the relationship between channels. We introduce a new architectural unit, which we term the Squeeze-and- Excitation (SE) block, with the goal of improving the quality of representations produced by a network by explicitly modelling the interdependencies between the channels of its convolutional features. To this end, we propose a mechanism that allows the network to perform feature recalibration, through which it can learn to use global information to selectively emphasise informative features and suppress less useful ones.”)
As per Claim 12, the combination of Kingma, Yoshida, Chollet, Ioffe, and Ramachandran teaches the method of claim 11. Chollet teaches wherein the residual cell further comprises a third BN layer with a second [Swish] activation function, a second convolutional layer following the third BN layer, a fourth BN layer following the second convolutional layer, and a [squeeze and excitation (SE)] layer following the fourth BN layer. (Chollet, Page 1255 Figure 5, as shown above in the rejection to Claim 10, discloses repeating the sequences of layers in the cell 3 times, and one of ordinary skill in the art will appreciate that one can stack these layers any number of times, thus resulting in second, third, fourth layers, etc.)
However, Chollet does not teach Swish activation function; squeeze and excitation (SE) layer
Ramachandran teaches Swish activation function (Ramachandran, Top of Page 2, discloses: “The best discovered activation function, which we call Swish, is f(x) = x sigmoid(x), where is a constant or trainable parameter.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ramachandran with Kingma, Yoshida, Chollet, and Ioffe for at least the reasons recited in the rejection to claim 10.
However, the combination of Kingma, Yoshida, Chollet, Ioffe, and Ramachandran does not teach squeeze and excitation (SE) layer
Hu teaches squeeze and excitation (SE) layer (Hu, Page 9 Right Column, Para 2, discloses: “We also construct a variant of the design which moves the SE block inside the residual unit, placing it directly after the 3 x 3 convolutional layer.”)
Hu is analogous art because it is in the field of endeavor of machine learning and convolutional neural networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, and Swish activation of Kingma, Chollet, Ioffe, and Ramachandran with the squeeze and excitation layer of Hu to be placed after the convolutional layer of the residual unit. One of ordinary skill in the art would be motivated to do so in order to improve the quality of results by focusing on the most informative features (Hu, Page 1 Right Column Para 1: “In this paper, we investigate a different aspect of network design - the relationship between channels. We introduce a new architectural unit, which we term the Squeeze-and- Excitation (SE) block, with the goal of improving the quality of representations produced by a network by explicitly modelling the interdependencies between the channels of its convolutional features. To this end, we propose a mechanism that allows the network to perform feature recalibration, through which it can learn to use global information to selectively emphasise informative features and suppress less useful ones.”)
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma and Yoshida, further in view of Chen et al. (“Training Deep Nets with Sublinear Memory Cost”; hereinafter “Chen”)
As per Claim 14, the combination of Kingma and Yoshida teaches the method of claim 4 as well as training the VAE (see rejection to claim 4). However, the combination does not teach wherein training the VAE comprises: storing a first subset of activations generated by the VAE from the training dataset during a forward pass associated with training the VAE; and recalculating a second subset of activations generated by the VAE based on the stored first subset of activations during a backward pass associated with training the VAE to reduce a memory consumption associated with training of the VAE.
Chen teaches wherein training the VAE comprises: storing a first subset of activations generated by the VAE from the training dataset during a forward pass associated with training the VAE; and recalculating a second subset of activations generated by the VAE based on the stored first subset of activations during a backward pass associated with training the VA, wherein the second subset of activations is determined based on reducing a memory consumption associated with training of the VAE. (Recall above that Kingma teaches a VAE. Chen, Middle of Page 3, discloses: “Allocation plan in Fig. 1 contains examples of both cases. The first sigmoid transformation is carried out using inplace operation to save memory, which is then reused by its backward operation.” Chen, Abstract, discloses: “We focus on reducing the memory cost to store the intermediate feature maps and gradients during training.”)
Chen is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the training of a VAE of Kingma with the gradient-checkpointing of Chen. One of ordinary skill in the art would be motivated to do so in order to reduce memory consumption (Chen, Abstract: “Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.”)
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma and Yoshida, further in view of Micikevicius et al. (“Mixed Precision Training”; hereinafter “Micikevicius”)
As per Claim 15, the combination of Kingma and Yoshida teaches the method of claim 4 as well as training the VAE (see rejection to claim 4). However, the combination does not teach wherein training the VAE comprises storing a first portion of the one or more parameters using a first precision and storing a second portion of the one or more parameters using a second precision that is lower than the first precision.
Micikevicius teaches wherein training the VAE comprises storing a first portion of the one or more parameters using a first precision and storing a second portion of the one or more parameters using a second precision that is lower than the first precision (Recall above that Kingma teaches a VAE. Micikevicius, Page 5 Section 4, discloses: “Mixed Precision (MP): FP16 is used for storage and arithmetic. Weights, activations and gradients are stored using in FP16, an FP32 master copy of weights is used for updates.”)
Micikevicius is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the training of a VAE of Kingma with the mixed precision training of Micikevicius. One of ordinary skill in the art would be motivated to do so in order to reduce memory consumption (Micikevicius, Abstract: “We introduce methodology for training deep neural networks using half-precision floating point numbers, without losing model accuracy or having to modify hyperparameters. This nearly halves memory requirements and, on recent GPUs, speeds up arithmetic.”)
Claims 16 and 18-22 are rejected under 35 U.S.C. 103 as being unpatentable over Kingma in view of Chollet et al. (“Xception: Deep Learning With Depthwise Separable Convolutions” hereinafter “Chollet”), further in view of Ioffe (“Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models”), and further in view of Nachum et al. (“US 2019/0147339 A1”; hereinafter “Nachum”). Examiner note: In these rejections both Chollet and Ioffe are recited to teach batch normalization. Kingma teaches residual blocks, and Chollet teaches that batch normalization be used after each convolution in a residual block. Meanwhile Ioffe teaches a particular variant of batch normalization that overcomes Kingma’s stated difficulties with it. Including Ioffe also offers consistency with the combination of references used for rejections in claims 5-8, 18, and 23-25, which explicitly recite Ioffe’s “momentum parameter”. Details are provided below.
As per Claim 16, Kingma teaches a method for performing machine learning, comprising:
inputting a training dataset into a variational autoencoder (VAE) comprising an encoder network, a prior network, and a decoder network (Kingma, Page 7 Section 6.2, discloses the use of images: “We also evaluated IAF on the CIFAR-10 dataset of natural images.”
Kingma, top of Page 2, discloses training: “In particular, we train deep variational auto-encoders with latent variables at multiple levels of the hierarchy.”
Examiner notes that a “variational auto-encoder” comprises an encoder neural network and a decoder. Kingma also explicitly discloses these components at the bottom of Page 13: “The first layer of the encoder is a convolutional layer with 2 × 2 spatial subsampling; the last layer of the decoder has a matching convolutional layer with 2 × 2 spatial upsampling.” The encoder and decoder are neural networks as per Kingma, Page 1 Intro Para 1: “When using neural networks for both the inference network and generative model, this results in class of models called variational autoencoders (Kingma and Welling, 2013) (VAEs).” One of ordinary skill in the art will appreciate that the inference network is the encoder and the generative model is the decoder.
As for the prior, this is also a neural network. Note that Kingma on Page 7 Figure 3, as shown below, discloses a “Layer Prior”:
PNG
media_image4.png
236
400
media_image4.png
Greyscale
Furthermore, note that Kingma Page 11 Figure 4 gives further detail:
PNG
media_image5.png
338
325
media_image5.png
Greyscale
Note above that Kingma discloses “parameters of p(z|.)”, which is parameters of the prior distribution, which is then sampled from. Kingma, Top of Page 11, states: “Assuming L layers of latent variables, the generative model’s density function is factorized as p(x, z1, z2, z3, ...) = p(x, z1:L) = p(x|z1:L)p(z1:L). The second part of this density, the prior over the latent variable, is autoregressive: p(z1:L) = p(zL) ПL−1l=1 p(zl|zl+1:L). This autoregressive nature of the prior increases the flexibility of the true posterior, leading to improved empirical results. This improved performance is easily explained: as the true posterior is largely a function of this prior, a flexible prior improves the flexibility of the true posterior, making it easier for the VAE to match the approximate and true posteriors, leading to a tighter bound, without sacrificing the flexibility of the generative model itself.” Thus, Kingma discloses a “prior” with learned “parameters” that is “autoregressive”, and thus a neural network, as Kingma states in the Abstract: “The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network.” Furthermore, Kingma also discloses at the bottom of Page 7: “We sampled from the PixelCNN naïvely by sequentially generating a pixel at a time, using the full generative model at each iteration.” Thus, Kingma suggests sampling from the prior is achieved by sampling from a neural network (PixelCNN)).
Thus, Kingma discloses an encoder neural network, prior neural network, and decoder neural network.
training the VAE by updating one or more parameters of the VAE based on [regularization of a scaling parameter associated with batch] normalization of one or more layers of the VAE (Kingma, Page 7 Section 6.2, discloses the use of images: “We also evaluated IAF on the CIFAR-10 dataset of natural images.” Kingma, top of Page 2, discloses training: “In particular, we train deep variational auto-encoders with latent variables at multiple levels of the hierarchy.” Kingma, Top of Page 4 Algorithm 1, discloses: “neural network parameters”.
Kingma also discloses to apply normalization, but not batch normalization (“We also found that the noise introduced by batch normalization hurts performance; instead we use weight normalization (Salimans and Kingma, 2016) method”).
Examiner notes that another reference (Chollet) will teach batch normalization. And another reference (Ioffe) will be applied that will describe how to alleviate the “hurt performance” for batch normalization.)
producing generative output that reflects a first distribution of the training dataset by applying the decoder network to one or more values sampled from a second distribution of latent variables generated by the prior network (Kingma, Page 11 Figure 4, discloses:
PNG
media_image5.png
338
325
media_image5.png
Greyscale
As shown above, Kingma discloses a layer that produces a new image (“Generative”) layer that is based on a visual attributes associated with the set of training images (“conv”, “slice features”), by applying the decoder portion (“Generative”) layer to a value generates based on an output of the prior (a “sample” from the prior p(z|.)”). Furthermore, Kingma also discloses at the bottom of Page 7: “We sampled from the PixelCNN naïvely by sequentially generating a pixel at a time, using the full generative model at each iteration.” Thus, Kingma suggests sampling from the prior is achieved by sampling from a neural network (PixelCNN).
Furthermore, Kingma discloses in Algorithm 1:
PNG
media_image6.png
32
279
media_image6.png
Greyscale
And thus discloses that the encoder produces a first distribution parameterized by
PNG
media_image7.png
20
44
media_image7.png
Greyscale
. Thus, the “prior” distribution disclosed above is a second distribution, that reflects the first distribution, and values are sampled from this second distribution (the prior), the prior being a distribution of latent variables (“Assuming L layers of latent variables”)).
However, Kingma does not teach based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Chollet teaches a plurality of batch normalization layers (Chollet, Page 1255 Figure 5 caption, discloses: “Note that all Convolution and SeparableConvolution layers are followed by batch normalization [7] (not included in the diagram)”.
Chollet is analogous art because it is in the field of endeavor of machine learning and deep residual networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual blocks and spectral norm regularization of Kingma, Yoshida with the batch normalization of Chollet. Chollet teaches to perform batch normalization after every convolutional layer, and in combination with Kingma’s decoder network which comprises convolutional networks, this would result in applying batch normalization to layers of the decoder network. One of ordinary skill in the art will appreciate that batch normalization is very well known in the art, and one would be motivated to do so in order to improve training performance, as indicated by one of the inventors of batch normalization, Ioffe, who is also cited below for momentum parameter (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models.”) As for Kingma’s stated difficulties regarding it, this problem of the combination will be solved by further combination with Ioffe shown below.
Ioffe also teaches a plurality of batch normalization layers (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”. Ioffe, Page 4 Section 3.1: “To Batch-Normalize a network, we specify a subset of activations and insert the BN transform for each of them, according to Alg. 1. Any layer that previously received x as the input, now receives BN(x))”).
Ioffe is analogous art because it is in the field of endeavor of machine learning and batch normalization (“Batch Renormalization”). It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, Swish activation, and S-E layers of Kingma, Chollet, Ramachandran, and Hu with the batch renormalization with “moving average update rate α” of Ioffe. While Chollet discloses “note that all Convolution and SeparableConvolution layers are followed by batch normalization”, Kingma notes that “we also found that the noise introduced by batch normalization hurts performance”. Ioffe’s “moving average update rate α” alleviates this problem of “noise” by smoothing out the average into a moving average. One of ordinary skill in the art would be motivated to adjust Ioffe’s update rate in order to gain the advantages of batch normalization (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models”), while avoiding the problems of noise in the minibatches (Ioffe, Abstract: “Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches.”) Thus, one of ordinary skill in the art would understand that adjusting this parameter would alleviate Kingma’s concerns about batch normalization, rendering the combination also with Chollet (who also teaches batch normalization), obvious.
However, the combination of Kingma, Chollet, and Ioffe does not teach based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Nachum teaches based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers (Examiner here is interpreting the word “added” not in the strict sense as “an addend of a sum”, but as an extra element that is included in the calculation, consistent with the terminology “apply” used in Specification [0046] and [0088]. Nachum directly applies a regularization to the batch norm scaling parameter as stated in [0007]: “a batch normalization regularization term comprising a sparsity-inducing norm of a scale parameter of a batch normalization layer of the neural network.” Furthermore, Nachum states in [0069]: “An example of a shrinking engine loss function is given by:
PNG
media_image13.png
300
485
media_image13.png
Greyscale
where N is the number of training examples in the training data, loss(w, xi, yi) is a task loss function, |w|22 is a L2 regularization loss on the weights of the neural network, J refers to the number of neurons to which batch normalization scale parameter regularization is applied, |γj|1 refers to the L1 norm of the batch normalization scale parameter of a particular neuron indexed by j, |wk|2 refers to a group lasso regularization term on the weights of the input connections of a particular neuron indexed by k (e.g., the L2 norm of the weights of the input connections of the particular neuron), and the λ-factors (i.e., λ1, {λ2j}j = 1J, and {λ3k}k=1K) are weighting factors.”
Here, Nachum teaches regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter (“batch normalization regularization term comprising a sparsity-inducing norm of a scale parameter of a batch normalization layer of the neural network”), where a regularization term (λ2j) is applied to a scaling parameter (γ). Nachum also teaches the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE (“|γj|1 refers to the L1 norm of the batch normalization scale parameter”). Examiner also notes that Chollet and Ioffe above disclose a plurality of batch normalization layers, and therefore the combination teaches the regularization term being based on a norm of a plurality of scaling parameters associated with a plurality of batch normalization layers. Therefore, when taking into account that this is done for a plurality of batch normalization layers, a norm is taken of a plurality of scaling factors, wherein there is a scaling factor for each batch normalization. As for the limitation each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers, since this is performed on a plurality of batch normalization layers, and each batch normalization only has one scaling factor gamma, then each scaling factor is associated with a different batch normalization layer.)
Nachum is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Kingma, Yoshida, Chollet, and Ioffe with Nachum. One of ordinary skill in the art would have been motivated to perform regularization on the batch normalization scaling parameter as known in the art via previous work by Luo and van Laarhoven, as it serves to improve the learning rate (Luo Page 9 Conclusion: “This work investigated an explicit regularization form of BN, which was decomposed into PN and gamma decay where the regularization strengths from B and B were explored. Moreover, optimization and generalization of BN with regularization were derived and compared with vanilla SGD, WN, and WN+gamma decay, showing that BN enables training to converge with large maximum and effective learning rate, as well as leads to better generalization.”) Furthermore, Luo cites Van Laarhoven (“Towards Understanding Regularization in Batch Normalization”) on Page 2 (“Moreover, van Laarhoven (2017) showed that weight decay has no regularization effect when using together with BN or WN”) and Page 5 (“The effective LRs shown in Table 1 are consistent with previous work (van Laarhoven, 2017.))” Examiner notes that Van Laarhoven already suggested regularizing the scaling parameter (Van Laarhoven, Top of Page 2, “In this paper we investigate the effects of L2 regularization in combination with Batch, Weight and Layer Normalization. We show that, as expected, there is no regularizing effect. Rather, the ‘regularization’ term strongly influences the learning rate” and Page 3 Above Section 4, “We can now also answer the question of why L2 regularization is still beneficial when training neural networks with Batch Normalization: If no regularization is used the weights can grow unbounded, and the effective learning rate goes to 0.”) Furthermore, Van Laarhoven, Bottom of Page 8 states: “With batch normalization we have added two additional parameters, γ and β, and it of course makes sense to also regularize these. In our experiments we did not use regularization for these parameters, though preliminary experiments show that regularization here does not affect the results.”)
As per Claim 18, the combination of Kingma, Chollet, Ioffe, and Nachum teaches the non-transitory computer readable medium of claim 16. Ioffe teaches wherein applying the decoder network to the one or more values comprises applying the batch normalization to the one or more layers based on a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization. (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”.)
As per Claim 19, the combination of Kingma, Chollet, Ioffe, and Nachum teaches the non-transitory computer readable medium of claim 16. Kingma teaches wherein the VAE comprises a hierarchy of groups of the latent variables, and wherein a first sample from a first group in the hierarchy is combined with a feature map and passed to a second group following the first group in the hierarchy for use in generating a second sample from the second group (Kingma, Page 5 Figure 2, discloses:
PNG
media_image11.png
228
931
media_image11.png
Greyscale
As shown above, Kingma discloses a hierarchy of groups of latent variables (µ, σ, h), where a first sample (z) is combined with a feature map (h) and passed to a second group (“IAF Step”) for use in generating a second sample (z)).
As per Claim 20, the combination of Kingma, Chollet, Ioffe, and Nachum teaches the non-transitory computer readable medium of claim 19. Kingma teaches wherein the encoder network comprises a bottom-up model and a top-down model that perform bidirectional inference of the groups of the latent variables based on the training dataset. (Recall above that Kingma teaches latent variables based on the training dataset as shown in the rejection to claim 19. Kingma, Bottom of Page 10, also discloses: “Our architecture consists of L stacked blocks, where each block (l = 1..L) is a combination of a bottom-up residual unit for inference, producing a series of bottom-up activations h(q) l , and a top-down residual unit used for both inference and generation, producing a series of top-down activations.” Here, Kingma discloses that the encoder (“inference”) comprises a bottom-up and top-down model, and thus bidirectional inference.)
As per Claim 21, the combination of Kingma, Chollet, Ioffe, and Nachum teaches the non-transitory computer readable medium of claim 20. Recall above that Kingma teaches hierarchy of groups of the latent variables as shown in the rejection to claim 19. Kingma teaches wherein producing the generative output comprises: executing the top-down model to sample the one or more values along the hierarchy of groups of the latent variables (Kingma, as shown above in the rejection to claim 20, discloses a “top-down” residual unit for “generation”, which is the decoder.)
inputting the sampled one or more values into the decoder network to produce the generative output (Kingma, as shown above in the rejection to claim 16, discloses sampling from the prior in the decoder network to produce generative output.)
As per Claim 22, the combination of Kingma, Chollet, Ioffe, and Nachum teaches the non-transitory computer readable medium of claim 16 as well as applying the decoder network to the one or more values and one or more values sampled from the second distribution (see Kingma in rejection to claim 16). Ioffe teaches wherein applying the decoder network to the one or more values comprises recalculating batch statistics associated with the batch normalization based on the one or more values sampled from the second distribution (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”.)
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kingma, Chollet, Ioffe, and Nachum further in view of Yoshida.
As per Claim 17, the combination of Kingma, Chollet, Ioffe, and Nachum teaches the non-transitory computer readable medium of claim 16. However, the combination does not teach wherein training the VAE further comprises updating the one or more parameters of the VAE based on an objective function comprising a spectral regularization term that controls a smoothness of one or more outputs produced by the VAE from the training dataset.
Yoshida teaches updating the one or more parameters of the VAE based on an objective function comprising a spectral regularization term that controls a smoothness of one or more outputs produced by the VAE from the training dataset. (Yoshida, Page 3 Section 3.2, discloses: “In this subsection, we explain spectral norm regularization. The notations are the same as those used in Section 3.1. To bound the spectral norm of each weight matrix, W`, we consider the following empirical risk minimization problem
PNG
media_image8.png
66
586
media_image8.png
Greyscale
where
PNG
media_image9.png
20
58
media_image9.png
Greyscale
is a regularization factor. We refer to the second term as the spectral norm regularizer. It decreases the spectral norms of the weight matrices. Here, Yoshida’s Eq 1 discloses an objective function that includes spectral regularization term (“second term as the spectral norm regularizer”).
Yoshida further discloses in Page 2 Section 3: “In this section, we explain spectral norm regularization and how it reduces the sensitivity to test data perturbation.” Here, Yoshida discloses that the output data is less “sensitive” to “perturbation” in input data, and therefore the outputs are smoother. Yoshida, at the top of Page 5, describes this as “smoothness” when discussing competing approaches to mitigating perturbation sensitivity: “smoothness of a model against input perturbation.”)
Yoshida is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder of Kingma with the spectral norm regularization of Yoshida. One of ordinary skill in the art would be motivated to do so in order to achieve better generalizability of results, by reducing sensitivity to perturbations (Yoshida, Page 8 Section 5 Conclusion: “In this work, we hypothesized that a high sensitivity to the perturbation of the input data degrades the performance of the data. In order to reduce the sensitivity to the perturbation of the test data, we proposed the spectral norm regularization method, and confirmed that it exhibits a better generalizability than other baseline methods through experiments. Experimental comparison with other methods indicated that the insensitivity to the perturbation of the test data plays a crucial role in determining the generalizability.”)
Claims 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Kingma et al. (“Improved variational inference with inverse autoregressive flow”; hereinafter “Kingma”) in view of Chollet et al. (“Xception: Deep Learning With Depthwise Separable Convolutions” hereinafter “Chollet”), further in view of Ioffe (“Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models”), further in view of Ramachandran et al. (“Searching for Activation Functions”; hereinafter “Ramachandran”), further in view of Hu et al. (“Squeeze-and-Excitation Networks”; hereinafter “Hu”), further in view of Chen, and further in view of Nachum.
As per Claim 23, Kingma teaches A system, comprising: a memory that stores instructions, and a processor that is coupled to the memory and, when executing the instructions, is configured to: (Kingma, Bottom of Page 7, discloses the use of a computer system (“GPU”, which is a processor), and thus suggests the use of memory and a processor: “Sampling took about 0.05 seconds/image with the ResNet VAE model, versus 52.0 seconds/image with the PixelCNN model, on a NVIDIA Titan X GPU.”)
sample one or more values from a first distribution of latent variables associated with an encoder network included in a variational autoencoder (VAE) (Kingma, Top of Page 2, discloses a variational auto-encoder: “We demonstrate this method by improving inference networks of deep variational auto-encoders. In particular, we train deep variational auto-encoders with latent variables at multiple levels of the hierarchy, where each stochastic variable is a three-dimensional tensor (a stack of feature maps), and demonstrate improved performance.” Kingma, Page 5 Figure 2, discloses:
PNG
media_image14.png
155
532
media_image14.png
Greyscale
Above, one sees that the Encoder learns the parameters of a first distribution (σ, µ), and then a sample “z” is taken from this distribution. Also see Kingma Page 4 Algorithm 1, which also explains that z is sampled from a first distribution of latent variables from the encoder:
PNG
media_image15.png
130
560
media_image15.png
Greyscale
)
wherein the encoder network comprises a first residual cell (Kingma, Bottom of Page 10, discloses: “For CIFAR-10, we used a novel neural variational autoencoder (VAE) architecture with ResNet (He et al., 2015, 2016) units and multiple stochastic layers. Our architecture consists of L stacked blocks, where each block (l = 1..L) is a combination of a bottom-up residual unit for inference, producing a series of bottom-up activations h(q) l , and a top-down residual unit used for both inference and generation, producing a series of top-down activations h(p) l . The hidden layer of each residual function in the generative model contains a combination of the usual deterministic hidden units and a relatively small number of stochastic hidden units with a heteroscedastic diagonal Gaussian distribution p(zl|h(p) l ) given the unit’s input h(p) l , followed by a nonlinearity. We utilize wide (Zagoruyko and Komodakis, 2016) pre-activation residual units (He et al., 2015) with single-hidden-layer residual functions.”
Here, Kingma discloses a first residual cell (“residual unit”, “hidden layer of each residual function in the generative model contains a combination of the usual deterministic hidden units”, “pre-activation residual units”)).
sample from the second distribution to produce generative output associated with the data (Kingma, Page 11 Figure 4, discloses:
PNG
media_image5.png
338
325
media_image5.png
Greyscale
As shown above, Kingma discloses a layer that produces a new image (“Generative”) layer that is based on a visual attributes associated with the set of training images (“conv”, “slice features”), by applying the decoder portion (“Generative”) layer to a value generates based on an output of the prior (a “sample” from the prior p(z|.)”). Furthermore, Kingma also discloses at the bottom of Page 7: “We sampled from the PixelCNN naïvely by sequentially generating a pixel at a time, using the full generative model at each iteration.” Thus, Kingma suggests sampling from the prior is achieved by sampling from a neural network (PixelCNN).
Furthermore, Kingma discloses in Algorithm 1:
PNG
media_image6.png
32
279
media_image6.png
Greyscale
And thus discloses that the encoder produces a first distribution parameterized by
PNG
media_image7.png
20
44
media_image7.png
Greyscale
. Thus, the “prior” distribution disclosed above is a second distribution, that reflects the first distribution, and values are sampled from this second distribution (the prior), the prior being a distribution of latent variables (“Assuming L layers of latent variables”)).
apply [batch] normalization to one or more layers of a decoder network included in the VAE based on (i) the one or more values [and (ii) a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization] to generate parameters of a second distribution of data with which the VAE is trained (As shown above, Kingma teaches that a decoder network (“generative layer”) generates parameters of a second distribution of data (“prior”) based on the values sampled from the first distribution. Kingma also discloses to apply normalization, but not batch normalization (“We also found that the noise introduced by batch normalization hurts performance; instead we use weight normalization (Salimans and Kingma, 2016) method”).
Examiner notes that another reference (Chollet) will teach batch normalization. And another reference (Ioffe) will be applied that will describe how to alleviate the “hurt performance” for batch normalization.)
However, Kingma does not teach wherein the first residual cell comprises a first batch normalization (BN) layer fused with a first Swish activation function, a first convolutional layer following the first BN layer fused with the first Swish activation function, a second BN layer fused with a second Swish activation function, a second convolutional layer following the second BN layer fused with the second Swish activation function, and a first squeeze and excitation (SE) layer following the second convolutional layer; apply batch normalization … based on … (ii) a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization; wherein applying the batch normalization comprises updating one or more parameters of the VAE based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Chollet teaches plurality of batch normalization layers; wherein the first residual cell comprises a first batch normalization (BN) layer [fused] with a first [Swish] activation function, a first convolutional layer following the first BN layer [fused] with the first [Swish] activation function, a second BN layer [fused] with a second [Swish] activation function, a second convolutional layer following the second BN layer [fused] with the second [Swish] activation function (Chollet, Page 1255 Figure 5, discloses:
PNG
media_image12.png
642
1146
media_image12.png
Greyscale
Chollet, as shown above, discloses that “Note that all Convolution and SeparableConvolution layers are followed by batch normalization [7] (not included in the diagram)”, and thus Examiner has indicated where they would appear in the diagram (after convolutions, before ReLU). Furthermore, Chollet, discloses the use of residual cells in Page 1253 Section 3 Para 2: “The 36 convolutional layers are structured into 14 modules, all of which have linear residual connections around them, except for the first and last modules.”)
Chollet is analogous art because it is in the field of endeavor of machine learning and deep convolutional neural networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder of Kingma with the depthwise separable convolutions and batch normalization of Chollet, by incorporating Chollet’s architecture of residual blocks into the residual blocks of Kingma. One of ordinary skill in the art would have been motivated to do so in order to gain increased performance at a low cost of effort (Chollet, Page 1257, Conclusion: “Compared to Inception V3, Xception shows small gains in classification performance on the ImageNet dataset and large gains on the JFT dataset. We expect depthwise separable convolutions to become a cornerstone of convolutional neural network architecture design in the future, since they offer similar properties as Inception modules, yet are as easy to use as regular convolution layers.”) Examiner further points out that depthwise separable convolutions have become well known in the art for their efficiency, as for example, another reference Sandler et al. (“MobileNetV2: Inverted Residuals and Linear Bottlenecks”) discloses in Page 4511 Section 3.1: “Depthwise Separable Convolutions are a key building block for many efficient neural network architectures [27, 28, 20] and we use them in the present work as well … Effectively depthwise separable convolution reduces computation compared to traditional layers by almost a factor of k2.” Furthermore, Chollet teaches to perform batch normalization after every convolutional layer, and in combination with Kingma’s decoder network which comprises convolutional networks, this would result in applying batch normalization to layers of the decoder network. One of ordinary skill in the art will appreciate that batch normalization is very well known in the art, and one would be motivated to do so in order to improve training performance, as indicated by one of the inventors of batch normalization, Ioffe, who is also cited below for momentum parameter (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models.”) As for Kingma’s stated difficulties regarding it, this problem of the combination will be solved by further combination with Ioffe shown below.
However, the combination of Kingma and Chollet does not teach Swish activation function; a first squeeze and excitation (SE) layer following the second convolutional layer; apply batch normalization … based on … (ii) a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization; BN layer fused with an activation function; convolutional layer following the BN layer fused with the activation function; wherein applying the batch normalization comprises updating one or more parameters of the VAE based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Ioffe teaches plurality of batch normalization layers; apply batch normalization … based on … (ii) a momentum parameter that increases a rate at which a running statistic associated with the batch normalization catches up to a batch statistic associated with the batch normalization (Ioffe, Top of Page 4, Algorithm 1, discloses: “Values of x over a training mini-batch B = {x1…m}; parameters γ,β; current moving mean µ and standard deviation σ; moving average update rate α”. Ioffe, Page 4 Section 3.1: “To Batch-Normalize a network, we specify a subset of activations and insert the BN transform for each of them, according to Alg. 1. Any layer that previously received x as the input, now receives BN(x))”).
Ioffe is analogous art because it is in the field of endeavor of machine learning and batch normalization (“Batch Renormalization”). It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions and batch normalization layers of Kingma and Chollet with the batch renormalization with “moving average update rate α” of Ioffe. While Chollet discloses “note that all Convolution and SeparableConvolution layers are followed by batch normalization”, Kingma notes that “we also found that the noise introduced by batch normalization hurts performance”. Ioffe’s “moving average update rate α” alleviates this problem of “noise” by smoothing out the average into a moving average. One of ordinary skill in the art would be motivated to adjust Ioffe’s update rate in order to gain the advantages of batch normalization (Ioffe, Abstract: “Batch Normalization is quite effective at accelerating and improving the training of deep models”), while avoiding the problems of noise in the minibatches (Ioffe, Abstract: “Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches.”) Thus, one of ordinary skill in the art would understand that adjusting this parameter would alleviate Kingma’s concerns about batch normalization, rendering the combination also with Chollet (who also teaches batch normalization), obvious.
However, the combination of Kingma, Chollet, and Ioffe does not teach Swish activation function; a first squeeze and excitation (SE) layer following the second convolutional layer; BN layer fused with an activation function; convolutional layer following the BN layer fused with the activation function; wherein applying the batch normalization comprises updating one or more parameters of the VAE based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Ramachandran teaches Swish activation function (Ramachandran, Top of Page 2, discloses: “The best discovered activation function, which we call Swish, is f(x) = x sigmoid(x), where is a constant or trainable parameter.”)
Ramachandran is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, and ReLU activation of Kingma, Chollet, and Ioffe with the Swish activation of Ramachandran, by simply replacing the ReLU used by Chollet with the Swish of Ramachandran. One of ordinary skill in the art would be motivated to do so in order to gain improved accuracy (Ramachandran, Top of Page 2: “Our extensive experiments show that Swish consistently matches or outperforms ReLU on deep networks applied to a variety of challenging domains such as image classification and machine translation. On ImageNet, replacing ReLUs with Swish units improves top-1 classification accuracy by 0.9% on Mobile NASNet-A (Zoph et al., 2017) and 0.6% on Inception-ResNet-v2 (Szegedy et al., 2017). These accuracy gains are significant given that one year of architectural tuning and enlarging yielded 1.3% accuracy improvement going from Inception V3 (Szegedy et al., 2016) to Inception-ResNet-v2 (Szegedy et al., 2017).”)
However, the combination of Kingma, Chollet, Ioffe, and Ramachandran does not teach a first squeeze and excitation (SE) layer following the second convolutional layer; BN layer fused with an activation function; convolutional layer following the BN layer fused with the activation function; wherein applying the batch normalization comprises updating one or more parameters of the VAE based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Hu teaches a first squeeze and excitation (SE) layer following the second convolutional layer (Hu, Page 9 Right Column, Para 2, discloses: “We also construct a variant of the design which moves the SE block inside the residual unit, placing it directly after the 3 x 3 convolutional layer.”)
Hu is analogous art because it is in the field of endeavor of machine learning and convolutional neural networks. It would have been obvious before the effective filing date of the claimed invention to combine the variational autoencoder with residual units comprising depthwise separable convolutions, batch normalization, and Swish activation of Kingma, Chollet, Ioffe, and Ramachandran with the squeeze and excitation layer of Hu to be placed after the convolutional layer of the residual unit. One of ordinary skill in the art would be motivated to do so in order to improve the quality of results by focusing on the most informative features (Hu, Page 1 Right Column Para 1: “In this paper, we investigate a different aspect of network design - the relationship between channels. We introduce a new architectural unit, which we term the Squeeze-and- Excitation (SE) block, with the goal of improving the quality of representations produced by a network by explicitly modelling the interdependencies between the channels of its convolutional features. To this end, we propose a mechanism that allows the network to perform feature recalibration, through which it can learn to use global information to selectively emphasise informative features and suppress less useful ones.”)
However, the combination of Kingma, Chollet, Ioffe, Ramachandran, and Hu does not teach BN layer fused with an activation function; convolutional layer following the BN layer fused with the activation function; wherein applying the batch normalization comprises updating one or more parameters of the VAE based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Chen teaches BN layer fused with an activation function; convolutional layer following the BN layer fused with the activation function (Chen, Middle of Page 3, discloses: “Allocation plan in Fig. 1 contains examples of both cases. The first sigmoid transformation is carried out using inplace operation to save memory, which is then reused by its backward operation.” Chen, Abstract, discloses: “We focus on reducing the memory cost to store the intermediate feature maps and gradients during training.” Here, Chen discloses fusing layers in order to use only a single feature map. Chen, Intro, discloses: “We mainly focus on reducing the memory cost to store intermediate results (feature maps) and gradients, as the size of the parameters are relatively small comparing to the size of the intermediate feature maps in many common deep architectures.” Chen specifically notes the usefulness of this architecture when applied to Chollet’s architecture on Page 5 Section 4.2: “One quick application of the general methodology is to drop the results of low cost operations and keep the results that are time consuming to compute. This is usually useful in a Conv-BatchNorm-Activation pipeline in convolutional neural networks. We can always keep the result of convolution, but drop the result of the batch normalization, activation function and pooling. In practice this will translate to a memory saving with little computation overhead, as the computation for both batch normalization and activation functions are cheap.”)
Chen is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the training of a VAE of Kingma with the gradient-checkpointing of Chen. One of ordinary skill in the art would be motivated to do so in order to reduce memory consumption (Chen, Abstract: “Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.”)
However, the combination of Kingma, Chollet, Ioffe, Ramachandran, Hu, and Chen does not teach wherein applying the batch normalization comprises updating one or more parameters of the VAE based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers
Nachum teaches wherein applying the batch normalization comprises updating one or more parameters of the VAE based on regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter, the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE, each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers. (Examiner here is interpreting the word “added” not in the strict sense as “an addend of a sum”, but as an extra element that is included in the calculation, consistent with the terminology “apply” used in Specification [0046] and [0088]. Nachum directly applies a regularization to the batch norm scaling parameter as stated in [0007]: “a batch normalization regularization term comprising a sparsity-inducing norm of a scale parameter of a batch normalization layer of the neural network.” Furthermore, Nachum states in [0069]: “An example of a shrinking engine loss function is given by:
PNG
media_image13.png
300
485
media_image13.png
Greyscale
where N is the number of training examples in the training data, loss(w, xi, yi) is a task loss function, |w|22 is a L2 regularization loss on the weights of the neural network, J refers to the number of neurons to which batch normalization scale parameter regularization is applied, |γj|1 refers to the L1 norm of the batch normalization scale parameter of a particular neuron indexed by j, |wk|2 refers to a group lasso regularization term on the weights of the input connections of a particular neuron indexed by k (e.g., the L2 norm of the weights of the input connections of the particular neuron), and the λ-factors (i.e., λ1, {λ2j}j = 1J, and {λ3k}k=1K) are weighting factors.”
Here, Nachum teaches regularization of a scaling parameter associated with batch normalization via a regularization term that is added to the scaling parameter (“batch normalization regularization term comprising a sparsity-inducing norm of a scale parameter of a batch normalization layer of the neural network”), where a regularization term (λ2j) is applied to a scaling parameter (γ). Nachum also teaches the regularization term being based on a norm value computed based on a plurality of scaling parameters associated with a plurality of batch normalization layers of the VAE (“|γj|1 refers to the L1 norm of the batch normalization scale parameter”). Examiner also notes that Chollet and Ioffe above disclose a plurality of batch normalization layers, and therefore the combination teaches the regularization term being based on a norm of a plurality of scaling parameters associated with a plurality of batch normalization layers. Therefore, when taking into account that this is done for a plurality of batch normalization layers, a norm is taken of a plurality of scaling factors, wherein there is a scaling factor for each batch normalization. As for the limitation each scaling parameter included in the plurality of scaling parameters being associated with a different batch normalization layer included in the plurality of batch normalization layers, since this is performed on a plurality of batch normalization layers, and each batch normalization only has one scaling factor gamma, then each scaling factor is associated with a different batch normalization layer.)
Nachum is analogous art because it is in the field of endeavor of machine learning. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Kingma, Yoshida, Chollet, and Ioffe with Nachum. One of ordinary skill in the art would have been motivated to perform regularization on the batch normalization scaling parameter as known in the art via previous work by Luo and van Laarhoven, as it serves to improve the learning rate (Luo Page 9 Conclusion: “This work investigated an explicit regularization form of BN, which was decomposed into PN and gamma decay where the regularization strengths from B and B were explored. Moreover, optimization and generalization of BN with regularization were derived and compared with vanilla SGD, WN, and WN+gamma decay, showing that BN enables training to converge with large maximum and effective learning rate, as well as leads to better generalization.”) Furthermore, Luo cites Van Laarhoven (“Towards Understanding Regularization in Batch Normalization”) on Page 2 (“Moreover, van Laarhoven (2017) showed that weight decay has no regularization effect when using together with BN or WN”) and Page 5 (“The effective LRs shown in Table 1 are consistent with previous work (van Laarhoven, 2017.))” Examiner notes that Van Laarhoven already suggested regularizing the scaling parameter (Van Laarhoven, Top of Page 2, “In this paper we investigate the effects of L2 regularization in combination with Batch, Weight and Layer Normalization. We show that, as expected, there is no regularizing effect. Rather, the ‘regularization’ term strongly influences the learning rate” and Page 3 Above Section 4, “We can now also answer the question of why L2 regularization is still beneficial when training neural networks with Batch Normalization: If no regularization is used the weights can grow unbounded, and the effective learning rate goes to 0.”) Furthermore, Van Laarhoven, Bottom of Page 8 states: “With batch normalization we have added two additional parameters, γ and β, and it of course makes sense to also regularize these. In our experiments we did not use regularization for these parameters, though preliminary experiments show that regularization here does not affect the results.”)
As per Claim 24, the combination of Kingma, Chollet, Ioffe, Ramachandran, Hu, Chen, and Nachum teaches the system of claim 23. Kingma teaches wherein the one or more values are sampled using a second residual cell (Kingma, as shown above in the rejection to claim 23, in Page 5 Figure 2 and Page 4 Algorithm 1, discloses sampling values from a distribution comprising an Encoder. Kingma also discloses residual cells in the bottom of Page 10: “Our architecture consists of L stacked blocks, where each block (l = 1..L) is a combination of a bottom-up residual unit for inference,” where the “inference” network comprises the encoder from which the distribution for sampling is generated, and Kingma discloses a second residual cell, because they disclose more than one (“each block…is a combination of a bottom-up residual unit for inference.”))
However, Kingma does not teach a second residual cell comprising a third BN layer, a third convolutional layer following the third BN layer, a fourth BN layer fused with a third Swish activation function, and a depthwise separable convolution layer following the fourth BN layer.
Cholet teaches a second residual cell comprising a third BN layer, a third convolutional layer following the third BN layer, a fourth BN layer fused with a third Swish activation function, and a depthwise separable convolution layer following the fourth BN layer. (Chollet, as shown above in the rejection to claim 23, discloses a first residual cell with a first and second BN layer, Swish activation, and convolutional layer. Chollet, Page 1255 Figure 5, also discloses “Repeated 8 times”, and thus discloses 8 residual cells, and therefore at least a second residual cell. Therefore Chollet also discloses a third BN layer, followed by a third convolutional layer, followed by a fourth BN layer, followed by a third Swish activation function, followed by another convolution layer. Regarding the convolution layers, Chollet discloses on Page 1253 Section 3: “We propose a convolutional neural network architecture based entirely on depthwise separable convolution layers.” Furthermore, on Figure 5, Chollet labels these layers “SeparableConv”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Chollet with Kingma for at least the reasons recited in the rejection to claim 23.
As per Claim 25, the combination of Kingma, Chollet, Ioffe, Ramachandran, Hu, Chen, and Nachum teaches the system of claim 24. Chollet teaches wherein the second residual cell further comprises a fifth BN layer fused with a fourth Swish activation function, a fourth convolutional layer following the fifth BN layer, a sixth BN layer following the fourth convolutional layer, and a second SE layer following the sixth BN layer. (Chollet, as shown above in the rejection to claim 23, discloses a first residual cell with a first and second BN layer, Swish activation, and convolutional layer. Chollet, Page 1255 Figure 5, also discloses “Repeated 8 times”, and thus discloses 8 residual cells, and therefore at least a second residual cell. Chollet also discloses repeating the sequences of layers in the cell 3 times, and one of ordinary skill in the art will appreciate that one can stack these layers any number of times, thus resulting in fourth, fifth, and sixth BN layers, swish activations, and convolutional layers.)
However, Chollet does not teach SE layer.
Hu teaches SE layer (Hu, Page 9 Right Column, Para 2, discloses: “We also construct a variant of the design which moves the SE block inside the residual unit, placing it directly after the 3 x 3 convolutional layer.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Hu with Chollet for at least the reasons recited in the rejection to claim 23.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEONARD A SIEGER whose telephone number is (571)272-9710. The examiner can normally be reached M-F 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached on (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEONARD A SIEGER/Examiner, Art Unit 2126