Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the application and claims filed 06/21/2024. 1-12, 14-21 are pending and have been examined. Claims 1-12, 14-21 are rejected.
Priority
The examiner acknowledges the priority benefit to U.S. Provisional Application No. 63/293,604, filed on December 23, 2021. The present application claims priority to U.S. Provisional Application No. 63/293,604, filed December 23, 2021.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/23/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim 4, 14, and 18 objected to because of the following informalities:
Claims 4 and 18 recite “the augmented representations a to generate” (remove “a”)
Claim 14 recite “One or non-transitory more computer-readable…” (should be “One or more”)
Claim 15 recites the limitation “A system comprising one or more computers… executed by one or more computers cause the one or more computers…” (Consider adding “executed by the one or more computers”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9-11 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitation "the training engine" There is insufficient antecedent basis for this limitation in the claim.
Claim 9 recites the limitation " the given training network input" There is insufficient antecedent basis for this limitation in the claim.
Claims 10-11 are rejected for being dependent on claim 9.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Examiner’s Note: Some rejections will include an Examiner’s Note (labeled ‘EN’) to provide additional context or rationale explaining the basis for the rejection.
Claim(s) 1, 4, and 5 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Alemi et al., "Deep Variational Information Bottleneck," (hereinafter Alemi).
Regarding claim 1, Alemi teaches:
A method performed by one or more computers, the method comprising (p. 17, "All of the networks for this paper were trained using TensorFlow (Abadi et al., 2016). ... these averaged parameters were used at test time."; References, p. 14, "Tensorflow: Large-scale machine learning on heterogeneous distributed systems." -- EN: Alemi builds and trains its networks in TensorFlow and uses them at test time. TensorFlow is machine learning software for computer systems. So Alemi's method is performed by one or more computers):
obtaining a network input (Appendix A, p. 17, "the 10,000 test images for test results. The input images were scaled to have values between -1 and 1 before fed to the network." -- EN: Alemi obtains each MNIST test image and feeds it to the network. Each test image is a network input); and
generating a final classification output for the network input using a neural network that comprises a neural network body and a neural network head, the generating comprising (Section 4.1, p. 5, "the stochastic encoder has the form p(z|x) = N(z|feμ(x), feΣ(x)), where fe is an MLP of the form 784 − 1024 − 1024 − 2K ... The decoder is a simple logistic regression model" -- EN: Alemi's network is the MLP fe followed by the decoder. The part of fe that outputs μ is the body. The decoder is the head; Section 4.1.1, p. 5, "We use a K = 256 dimensional bottleneck and a diagonal Gaussian for p(z|x)." -- EN: Section 4.1.1 uses this network with K = 256, and p(z|x) is a Gaussian. The network's output p(y|x), mapped below, is the final classification output):
processing the network input using the neural network body to generate an internal representation of the network input (Section 3, p. 4, "fe is an MLP which outputs both the K-dimensional mean μ of z"; Section 4.1, p. 5, "The first K outputs from fe encode μ" -- EN: the layers of fe that produce the first K outputs are the body. They take the 784 input values through the two 1024-unit layers. Their output μ, the mean of the Gaussian p(z|x), is the internal representation);
generating, from the internal representation of the network input, a plurality of augmented representations of the internal representation (Section 4.1.1, page 5-6, "zs ∼ p(z|x), where S = 12" -- EN: Alemi draws 12 samples zs from p(z|x), the Gaussian with mean μ quoted above; Section 3, p. 4, "z = f(x, ϵ) is a deterministic function of x and the Gaussian random variable ϵ" -- EN: each sample is computed from the input and random Gaussian noise ϵ. So each sample is a randomly varied copy of μ, and the 12 samples are augmented representations generated from μ);
-- EN: the instant specification's augmented representation denotes a representation in the same space as the internal representation but with different values, which can be sampled from a probability distribution (instant specification, Para. [0044], "Each augmented representation has the same dimensionality as the internal representation ... but has different values from the internal representation."; Para. [0050], "generate the augmented representations by sampling a fixed number of representations from the probability distribution"). Alemi's samples zs correspond to such representations.
processing each augmented representation using the neural network head to generate a respective initial classification output for each augmented representation (Section 4.1, p. 5, "The decoder is a simple logistic regression model of the form q(y|z) = S(y|fd(z)) ... fd(z) = Wz + b maps the K dimensional latent code to the logits of the C = 10 classes." -- EN: the decoder is the head. It maps a code z to scores for the 10 classes; Section 4.1.1, page 5-6, "Σs=1S q(y|zs)" -- EN: the head processes each sample zs. Each q(y|zs) is the initial classification output for that augmented representation); and
combining at least the respective initial classification outputs for the augmented representations to generate the final classification output for the network input (Section 4.1.1, page 5-6, "we average over 12 posterior samples (i.e., we use p(y|x) = (1/S) Σs=1S q(y|zs) for zs ∼ p(z|x), where S = 12)" -- EN: Alemi averages the 12 outputs q(y|zs) into p(y|x). This is the combining. p(y|x) is the final classification output).
Regarding claim 4, Alemi teaches all the limitations of claim 1, Alemi further teaches:
wherein combining at least the respective initial classification outputs for the augmented representations a to generate the final classification output for the network input comprises averaging the respective initial classification outputs for the augmented representations (Section 4.1.1, page 5-6, "we average over 12 posterior samples" -- EN: p(y|x) is the average of the 12 outputs q(y|zs)).
Regarding claim 5, Alemi teaches all the limitations of claim 1, Alemi further teaches:
wherein generating, from the internal representation of the network input, a plurality of augmented representations of the internal representation comprises:
processing the internal representation using an augmentation engine having augmentation parameters that have been learned through training to generate the plurality of augmented representations (Appendix A, p. 17, "For the 256 dimensional gaussian embeddings of Section 4.1.1, a linear layer of size 512 was used to create the 256 mean values and standard deviations for the embedding." -- EN: the embedding is a Gaussian. A layer creates its 256 mean values, which are μ, and its standard deviations σ. The weights that create σ, together with the step that draws the 12 samples zs from this Gaussian (as mapped for claim 1), are the augmentation engine. The engine centers the samples on μ. This is processing the internal representation to generate the augmented representations; Section 4.1.1, p. 5, "The networks were trained using TensorFlow for 200 epochs using the Adam optimizer" -- EN: the weights that create σ are trained with the network. They are the augmentation parameters learned through training).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Examiner’s Note: Some rejections will include an Examiner’s Note (labeled ‘EN’) to provide additional context or rationale explaining the basis for the rejection.
Claim(s) 2, 3, 14, 15, 16, 17, 18, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of Chadha et al. (US 2021/0150282 A1) (hereinafter Chadha).
Regarding claim 2, Alemi teaches all the limitations of claim 1.
Alemi teaches:
wherein the generating further comprises:
processing the internal representation using the neural network head to generate a respective initial classification output for the internal representation (Section 4.2.3, p. 9, "We also ran the VIB models in “mean mode”, where the σs are forced to be 0." -- EN: with the σs forced to 0, every sample z equals μ, so the decoder, which is the head, processes μ itself and outputs q(y|μ). This q(y|μ) is the initial classification output for the internal representation), wherein combining at least the respective initial classification outputs for the augmented representations to generate the final classification output for the network input comprises:
combining the respective initial classification outputs for the augmented representations (...) to generate the final classification output for the network input. (See the mapping above for claim 1.)
Alemi does not explicitly teach: [combining the respective initial classification outputs for the augmented representations] AND the respective initial classification output for the internal representation [to generate the final classification output for the network input]
EN: In essence, Alemi combines only the outputs for its augmented representations. Alemi does run the internal representation itself through the head in mean mode (Section 4.2.3, p. 9), but only as a separate test. All of its reported results use the 12-sample average (Section 4.2.3, p. 9, "all reported results are for stochastic evaluation with 12 samples"). So Alemi does not explicitly combine the output for the internal representation with the outputs for the augmented representations.
However, Chadha teaches:
[combining the respective initial classification outputs for the augmented representations] AND the respective initial classification output for the internal representation [to generate the final classification output for the network input] (See the entire Chadha Paragraph [0027] including “The two scaled versions and the original version are input into the machine learning model which detects objects such as text or tables and assigns confidence scores for each detected object. If objects are associated with overlapping bounding boxes (within each scale), then the system selects the highest score for a set of overlapping bounding boxes. Further, the range of confidence scores associated with each scaled image are normalized based on the range of the unscaled image. Each score may then be compared to a threshold and passing scores may be associated with objects that are identified as detected objects from the original input image.” – EN: "Each score" here includes the scores for the original image and for its two scaled copies ([0027], as quoted above). Chadha keeps the scores that pass a threshold and reports them together as the objects found in the original image. In essence, Chadha combines the original's output with the copies' outputs into one final result.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the test time augmentation of Chadha, which scores the original input together with its altered copies, because Chadha uses it "to provide a better chance of detecting an object or classifying the image" ([0023]), so the combination would be more accurate overall.
Regarding claim 3, Alemi in view of Chadha teaches all the limitations of claims 1 and 2, Alemi and Chadha further teach:
wherein combining the respective initial classification outputs for the augmented representations and the respective initial classification output for the internal representation to generate the final classification output for the network input comprises averaging the respective initial classification outputs for the augmented representations and the respective initial classification output for the internal representation (Alemi Section 4.1.1, page 5-6, "we average over 12 posterior samples" -- EN: Alemi averages the outputs for the samples; Chadha [0023], “The test image and the scaled test images may be input in the image detection model, various confidence scores may be assigned…”; Chadha [0012], "Generated confidence scores associated with the various manipulations of the augmented image may be averaged to obtain a final score for the image" -- EN: Chadha inputs the test image together with its scaled versions ([0023]; see also [0027]) and averages the scores obtained for the different versions into a final score ([0012]). In the combination set out for claim 2, the output for μ is averaged with the outputs for the 12 samples. This is the claimed averaging).
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the test time augmentation of Chadha, which scores the original input together with its altered copies, because Chadha uses it "to provide a better chance of detecting an object or classifying the image" ([0023]), so the combination would be more accurate overall.
Regarding claim 14, it is a non-transitory computer-readable storage media claim that recites substantially the same limitations as claim 1, and is therefore rejected under the same rationale as claim 1.
Alemi does not explicitly teach:
One or non-transitory more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: the operations set out above for claim 1.
(EN: Alemi runs its method on computers, as set out for claim 1. Alemi does not describe storage media that store the method as instructions.)
However, Chadha teaches:
One or non-transitory more computer-readable storage media storing instructions ([0129], "A non-transitory computer-readable medium storing code for image processing is described. The code may include instructions executable by a processor" -- EN: Chadha's medium storing the code denotes the storage media storing instructions) that when executed by one or more computers cause the one or more computers to perform operations comprising: the operations set out above for claim 1 ([0091], "The memory 625 may store computer-readable, computer-executable software including instructions that, when executed, cause the processor to perform various functions described herein." -- EN: the processor that executes the instructions denotes the one or more computers. In the combination, the operations are those set out for claim 1).
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the storage media of Chadha, which hold the method as instructions for a processor, because Chadha's system, run this way, "may be implemented in various types of computing systems, such as personal computers (including desktops, laptops, and mobile devices)" ([0093]), so the combination would be more versatile overall.
Regarding claim 15, it is a system claim that recites substantially the same limitations as claim 1, and is therefore rejected under the same rationale as claim 1.
Alemi does not explicitly teach:
A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: the operations set out above for claim 1.
(EN: Alemi runs its method on computers, as set out for claim 1. Alemi does not describe storage devices that store the method as instructions.)
However, Chadha teaches:
A system comprising one or more computers ([0087], "FIG. 6 shows a diagram of a system 600 including a device 605 ... memory 625, a processor 630" -- EN: the system 600 denotes the system. The device 605 with its processor 630 denotes the one or more computers) and one or more storage devices storing instructions ([0091], "The memory 625 may store computer-readable, computer-executable software including instructions" -- EN: the memory 625 denotes the storage devices storing instructions) that when executed by one or more computers cause the one or more computers to perform operations comprising: the operations set out above for claim 1 ([0092], "The processor 630 may be configured to execute computer-readable instructions stored in a memory 625 to perform various functions" -- EN: the functions the processor performs denote the operations. In the combination, the operations are those set out for claim 1).
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the processor and memory of Chadha, which store the method as instructions and run it, because Chadha's system, run this way, "may be implemented in various types of computing systems, such as personal computers (including desktops, laptops, and mobile devices)" ([0093]), so the combination would be more versatile overall.
Regarding claim 16, it is a system claim depending from independent claim 15 and recites substantially the same limitations as claim 2, and is therefore rejected under the same rationale as claim 2.
Regarding claim 17, it is a system claim depending from independent claim 15 and recites substantially the same limitations as claim 3, and is therefore rejected under the same rationale as claim 3.
Regarding claim 18, it is a system claim depending from independent claim 15 and recites substantially the same limitations as claim 4, and is therefore rejected under the same rationale as claim 4.
Regarding claim 19, it is a system claim depending from independent claim 15 and recites substantially the same limitations as claim 5, and is therefore rejected under the same rationale as claim 5.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of He et al., "Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack," (hereinafter PNI).
Regarding claim 6, Alemi teaches all the limitations of claims 1 and 5, Alemi further teaches:
wherein processing the internal representation using an augmentation engine having augmentation parameters that have been learned through training to generate the plurality of augmented representations comprises:
processing the internal representation using the augmentation engine and in accordance with the augmentation parameters … (Appendix A, p. 17, "For the 256 dimensional gaussian embeddings of Section 4.1.1, a linear layer of size 512 was used to create the 256 mean values and standard deviations for the embedding" -- EN: the augmentation engine, as mapped for claim 5, centers the Gaussian on the mean values μ and uses the standard deviations σ from its learned weights. This is processing μ with the engine in accordance with the augmentation parameters); and
sampling the plurality of augmented representations from the probability distribution (Section 4.1.1, page 5-6, "zs ∼ p(z|x), where S = 12" -- EN: Alemi samples the 12 augmented representations zs from p(z|x), the diagonal Gaussian mapped for claim 1. In the combination below, the engine generates the parameters of this Gaussian).
Alemi's engine uses μ unchanged as the mean. The only parameter it generates is σ, and it computes σ next to μ, not from μ (Section 4.1, p. 5, "The first K outputs from fe encode μ, the remaining K outputs encode σ"). So the engine does not process the internal representation to generate parameters of a probability distribution.
Alemi therefore does not explicitly teach:
processing the internal representation using the augmentation engine and in accordance with the augmentation parameters to generate parameters of a probability distribution over possible augmented representations;
However, PNI teaches:
processing the internal representation using the augmentation engine and in accordance with the augmentation parameters to generate parameters of a probability distribution over possible augmented representations (Section 3, p. 590, "The method that we propose to inject Gaussian noise to different components or locations within DNN can be mathematically described as: ṽl,i = fPNI(vl,i) = vl,i + αl · ηl,i; ηl,i ∼ N(0, σl2)" -- EN: PNI adds Gaussian noise, scaled by αl, to each element of a tensor vl. So the noisy tensor is drawn from a Gaussian centered on vl; Section 3, p. 590, "such vl can be input/weight/inter-layer (i.e., activation) tensor ... ηl shares the identical variance with v as in Eq. (5), thus the injected additive noise relies on αl and the distribution of vl simultaneously." -- EN: for an activation tensor, the noise variance is computed from the tensor itself and scaled by αl; Section 3, p. 590, "we set αl as a learnable parameter" -- EN: αl is a learned parameter);
-- EN: in the combination, Alemi's engine adds PNI's noise α·η to μ when it samples. Therefore, the engine computes this variance from μ, σ, and α. The weights that output σ and the learned α are the augmentation parameters. This is processing the internal representation using the engine, in accordance with the augmentation parameters, to generate parameters of a probability distribution over possible augmented representations.
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the trainable noise of PNI, which adds noise of a learned strength to the augmented representations, because PNI "improves both clean- and perturbed-data accuracy" (Abstract, p. 588), i.e., on both normal and attacked inputs, and Alemi already tests its networks against such attacks (Section 4.2, p. 6), so the combination would be more robust overall.
Claim(s) 7 and 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of Luo et al., "Few-Shot Learning via Feature Hallucination with Variational Inference," (hereinafter Luo).
Regarding claim 7, Alemi teaches all the limitations of claims 1 and 5 including “wherein processing the internal representation using an augmentation engine having augmentation parameters that have been learned through training to generate the plurality of augmented representations”.
Alemi draws its augmented representations zs directly from a Gaussian around μ, as mapped for claims 1 and 5. Alemi calls its head a decoder and calls z a latent code (Section 4.1, p. 5, "The decoder is a simple logistic regression model ... maps the K dimensional latent code to the logits"). But under the mapping of claim 1, the samples zs are the augmented representations, and Alemi's decoder is the head. Alemi has no encoder that maps μ to a latent distribution. It also has no decoder that turns a latent sample into an augmented representation.
Alemi therefore does not explicitly teach:
comprises:
processing the internal representation using an encoder neural network to generate parameters of a probability distribution over possible latent representations;
sampling a plurality of latent representations from the probability distribution; and
processing each latent representation using a decoder neural network to generate a respective augmented representation.
However, Luo teaches:
comprises:
processing the internal representation using an encoder neural network to generate parameters of a probability distribution over possible latent representations (Section 3.3.1, p. 3966, "we feed support features into E to establish distribution in L-dimensional latent space" -- EN: Luo's encoder E processes support features. Each such feature corresponds to the internal representation; Section 3.3.2, p. 3966, "The encoder is a simple fully-connected layer that uses support samples as input and directly outputs the distribution parameters (μc, σ2c) ∈ ℝL." -- EN: the encoder, a fully-connected layer, is the encoder neural network. Its output (μc, σ2c) is the set of parameters of a distribution over the latent space. In the combination, the encoder takes Alemi's μ as input);
sampling a plurality of latent representations from the probability distribution (Algorithm 2, p. 3967, "5: Sample n latent vectors {zj}nj=1 ∼ N(μc, σ2c) for each class" -- EN: the n latent vectors zj are the latent representations sampled from the distribution); and
processing each latent representation using a decoder neural network to generate a respective augmented representation (Section 3.3.2, p. 3966, "feature vectors can be directly generated by decoder D with latent vectors zic as input." -- EN: Luo's decoder D turns each latent vector into a feature vector. Each generated feature vector is an augmented representation; Section 4.1, p. 3969, "The encoder and decoder are both fully-connected layer" -- EN: the decoder, a fully-connected layer, is the decoder neural network. In the combination, the decoded vectors are in the space of μ. They replace Alemi's samples zs as the augmented representations that the head classifies).
-- EN: in the combination, the encoder, the sampling, and the decoder together are the augmentation engine of claim 5. The encoder and decoder weights are its augmentation parameters. Luo learns these weights through training (Algorithm 1, p. 3967, "20: Update E&D by minimizing Eq. 11").
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the encoder and decoder of Luo, which sample in a smaller latent space and decode each sample into an augmented representation, because Luo's encoder and decoder "generate diverse and effective samples that can make the decision boundary of the classifier sharper" (Section 3.2, p. 3965), i.e., augmented representations that improve the classifier, so the combination would be more accurate overall.
Regarding claim 8, Alemi in view of Luo teaches all the limitations of claims 1, 5, and 7, Luo further teaches:
wherein the encoder neural network and the decoder neural network have been trained jointly as a variational auto-encoder (VAE) (Luo Section 3.4, p. 3967, "minimize total loss to train E and D: Ltotal = Lcla + αLrec + βLlat" -- EN: one loss trains the encoder E and the decoder D together. This is joint training; Section 3.3.2, p. 3966, "the reconstruction loss between features decoded directly from the means vectors μi and the corresponding support samples" -- EN: Lrec is a reconstruction loss; Section 3.3.2, p. 3966, "Llat = 1/N Σc=1N KL(qϕ(zc|Sc)∥pθ(zc|Sc, Qc))" -- EN: Llat is a KL divergence loss; Section 3.3.3, p. 3966, "variational inference is a suitable approximating approach, which has been fully discussed in VAE [16][34]." -- EN: Luo trains E and D by the variational inference used in a VAE).
-- EN: the instant specification's VAE training uses a loss with a reconstruction term and a KL divergence term (instant specification, Para. [0085], "the VAE training loss can include, for each given augmentation representation, a reconstruction loss term ... and a regularization term, e.g., a KL divergence term, for the probability distribution over the latent space generated by the encoder of the VAE"). Luo's Lrec and Llat are such terms. Luo sets its prior from data instead of fixing it at N(0, 1) as a standard VAE does (Section 3.3.3, page 3966-3967). The claim does not require a fixed prior.
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the joint training of Luo, which trains the encoder and decoder together on one loss with three terms, because Luo reports that only with all three terms can its model "generate suitable features, which have similar distribution characteristics as support features" (Section 4.3, p. 3969), i.e., features spread like the real ones, so the combination would be more accurate overall.
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of Lenc et al., "Understanding image representations by measuring their equivariance and equivalence," (hereinafter Lenc).
Regarding claim 9, Alemi teaches all the limitations of claims 1 and 5.
Alemi discloses wherein the augmentation engine has been trained … (Section 4.1.1, p. 5, "The networks were trained using TensorFlow for 200 epochs using the Adam optimizer" -- EN: the trained network includes the augmentation engine as mapped for claim 5).
Alemi does not explicitly teach:
wherein the augmentation engine has been trained to minimize a loss that encourages, for a given training input, the training engine to generate augmented representations of the given training network input that are statistically similar to internal representations that would be generated by the neural network body for augmented inputs that have been augmented by applying data augmentation to the given training input.
-- EN: "the training engine" and "the given training network input" lack antecedent basis. They are read as "the augmentation engine" and "the given training input."
However, Lenc teaches:
wherein the augmentation engine has been trained to minimize a loss that encourages, for a given training input, the training engine to generate augmented representations of the given training network input that are statistically similar to internal representations that would be generated by the neural network body for augmented inputs that have been augmented by applying data augmentation to the given training input (Section 2.1, p. 3, "learning amounts to optimising the regularised reconstruction error E(Ag, bg) = λℛ(Ag) + (1/n) Σi=1n ℓ(φ(gxi), Agφ(xi) + bg)" -- EN: Lenc learns a map (Ag, bg) by minimizing the loss E. For each training image xi, the loss compares two vectors. Agφ(xi) + bg is what the map generates from the image's representation φ(xi). This is the augmented representation of the training input. φ(gxi) is the representation of the transformed image gxi. The loss pulls the first toward the second, which encourages them to be statistically similar; Section 2, p. 2, "in practice, in this paper we will focus on geometric transformations such as affine warps and flips of the image." -- EN: the transformation g is the data augmentation, and the transformed image gxi is the augmented input; Section 3.4, p. 8, "augmented with random image rotations" -- EN: Lenc itself calls such transformations augmentation).
-- EN: in the combination, Lenc's φ is Alemi's body, so φ(xi) is μ(xi). Alemi's augmentation engine includes one map for each rescaling. Each map is fit with Lenc's loss. The engine then draws the samples zs around the map's output for μ. So the samples are pulled toward μ(gxi), the body's representation of the rescaled image.
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the learned maps of Lenc, which predict the internal representation of each rescaled image from that of the original image, because, as Lenc notes, with such maps "the representation φ(x) needs to be computed just once" (Section 3.4, p. 8), i.e., the network body runs once, not once per rescaled image, so the combination would be more efficient overall.
Regarding claim 10, Alemi in view of Chadha in view of Lenc teaches all the limitations of claims 1, 5, and 9, Lenc further teaches:
wherein the augmentation engine has been trained after training of the neural network and while holding parameters of the neural network body fixed (Lenc Section 2.1, p. 3, "Given a representation φ and a transformation g, the goal is to find a mapping Mg satisfying (1). ... Estimating (Ag, bg) is naturally formulated as an empirical risk minimisation problem." -- EN: the representation φ is given in advance. Only the map (Ag, bg) is estimated. So the parameters that compute φ, the body, stay fixed while the map is trained; Section 3.2, p. 6, "using the ALEXN CNN [10] as a reference state-of-the-art deep feature extractor ... recovering most if not all the performance of the original classifier (43%)" -- EN: Lenc learns its maps for AlexNet, an existing trained classifier. So the maps are trained after the network).
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi in view of Chadha with the training approach of Lenc, which learns the maps for an already trained network without retraining that network, because maps learned this way end up "recovering most if not all the performance of the original classifier" (Section 3.2, p. 6), so the combination would keep the accuracy of the already trained network while avoiding the cost of retraining it.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of Chadha in view of Lenc in view of Guo et al., "On Calibration of Modern Neural Networks," (hereinafter Guo).
Regarding claim 11, Alemi in view of Chadha in view of Lenc teaches all the limitations of claims 1, 5, 9, and 10.
Lenc discloses wherein the augmentation engine has been trained … (Section 2.1, p. 3, "Given data x sampled from a set of natural images, learning amounts to optimising the regularised reconstruction error" -- EN: Lenc trains its maps on a set of images. In the combination set out for claims 9 and 10, the maps in the augmentation engine are trained this way).
Lenc optimizes its maps on ILSVRC training data (Section 3.2, p. 6, "in the ImageNet ILSVCR data (the reported error is measured on the validation data, but optimised on the training data)"). Lenc does not say that these inputs differ from the inputs used to train its network. So Lenc does not teach training the engine using training inputs that are different from the training inputs used to train the neural network.
Alemi in view of Chadha in view of Lenc therefore does not explicitly teach:
wherein the augmentation engine has been trained using training inputs that are different from training inputs in training data used to train the neural network.
However, Guo teaches:
wherein the augmentation engine has been trained using training inputs that are different from training inputs in training data used to train the neural network (Section 4, p. 4, "All methods are post-processing steps that produce (calibrated) probabilities. Each method requires a hold-out validation set" -- EN: Guo fits each component it adds to a trained network on a hold-out set. The hold-out inputs are different from the inputs used to train the network; Section 4.1, p. 5, "Parameters a and b can be optimized using the NLL loss over the validation set. It is important to note that the neural network’s parameters are fixed during this stage." -- EN: the added component is trained on the hold-out inputs while the network stays fixed).
-- EN: in the combination, the maps in the augmentation engine are fit on a hold-out set that is separate from Alemi's training data. The claim does not exclude the engine also having weights trained on Alemi's training data, such as the weights that output σ.
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi in view of Chadha in view of Lenc with Guo's practice of fitting parts added after training on a separate hold-out set, because, as Guo explains, "neural networks can overfit to NLL without overfitting to the 0/1 loss" (Section 3, p. 4), i.e., a network's outputs on its training inputs are a poor guide to new inputs, so the combination would be more reliable overall.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of Sun et al. (US 2020/0241514 A1) (hereinafter Sun).
Regarding claim 12, Alemi teaches all the limitations of claim 1 including “wherein processing each augmented representation using the neural network head to generate a respective initial classification output for each augmented representation”.
Alemi does not explicitly teach:
comprises processing the augmented representations in parallel.
However, Sun teaches:
comprises processing the augmented representations in parallel ([0043], "Since the forward passes involve a number of independent and fixed realizations of the RNN model distribution, they can be done concurrently" -- EN: Sun runs its independent Monte Carlo forward passes concurrently, that is, in parallel. In the combination, the head's passes on Alemi's 12 samples zs are also independent of one another. They are run in parallel in the same way).
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi with the concurrent forward passes of Sun, which, like Alemi's samples, are random passes through a network whose outputs are averaged, because, as Sun explains, such passes "can be done concurrently, thus making variational dropout a good candidate for online monitoring" ([0043]), i.e., fast enough for real-time use, so the combination would be more efficient overall.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of Chadha in view of PNI.
Regarding claim 20, it is a system claim depending from independent claim 15 and recites substantially the same limitations as claim 6, and is therefore rejected under the same rationale as claim 6.
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi in view of Chadha with the trainable noise of PNI, which adds noise of a learned strength to the augmented representations, because PNI "improves both clean- and perturbed-data accuracy" (Abstract, p. 588), i.e., on both normal and attacked inputs, and Alemi already tests its networks against such attacks (Section 4.2, p. 6), so the combination would be more robust overall.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Alemi in view of Chadha in view of Luo.
Regarding claim 21, it is a system claim depending from independent claim 15 and recites substantially the same limitations as claim 7, and is therefore rejected under the same rationale as claim 7.
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the classification method of Alemi in view of Chadha with the encoder and decoder of Luo, which sample in a smaller latent space and decode each sample into an augmented representation, because Luo's encoder and decoder "generate diverse and effective samples that can make the decision boundary of the classifier sharper" (Section 3.2, p. 3966), i.e., augmented representations that improve the classifier, so the combination would be more accurate overall.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAYMUR RAHMAN ALI whose telephone number is (571)272-0007. The examiner can normally be reached Mon-Fri. 9:30-6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NAYMUR RAHMAN ALI/ Examiner, Art Unit 2123
/ALEXEY SHMATOV/ Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights