DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/11/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6-8, 13-15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. US Patent (US 11710045 B2) in view of Hu et al. US PG-Pub(US 20240233077 A1).
Regarding Claim 1, Lee teaches a method comprising: training, using at least one processing device of an electronic device(FIG. 2 is a conceptual block diagram of a classifier engine in the system of FIG. 1,), a student variational autoencoder (VAE) based on a teacher VAE(Col 7, Lines 24-32, “The input dataset received in block 300 is also used to train the VAE 202 in block 308. In one embodiment, the VAE is trained to learn a representation of the input dataset in the latent space. Once trained, the VAE 202 may be used for generating augmented data in blocks 310 and 314. In this regard, in block 310, the classifier engine 115 invokes the trained VAE model to generate extra supplemental unlabeled data samples 312 in the latent space. The extra data samples may be generated using, for example, a random input..”, discloses training a student VAE using teacher data from another VAE to generate unlabeled data samples in a latent space.), wherein training the student VAE based on the teacher VAE comprises using a first optimizer and a second optimizer(Col 8, Lines 30-34, “FIG. 4 is a more detailed block diagram of the VAE 202 according to one exemplary embodiment. In general terms, the VAE is a neural network that may include an encoder network (simply referred to as an “encoder”) 400 and a decoder network (simply referred to as a “decoder” 402.”, discloses the VAE is trained based on an encoder and decoder which the specification of the instant application discloses the first optimizer to be an encoder and the second optimizer is a decoder.)
Lee does not explicitly teach wherein the first optimizer is configured to align a latent space of the student VAE with a latent space of the teacher VAE;and wherein the second optimizer is configured to optimize reconstruction performance of the student VAE.
Hu teaches wherein the first optimizer is configured to align a latent space of the student VAE with a latent space of the teacher VAE ([0022], “Further, a purpose of knowledge transfer is achieved by aligning respective inter-layer relationship matrices of teachers and students; and a specific processing flow of output features X and Y of different layers is as follows”, ¶[0022]-¶[0024] disclose a process of using knowledge distillation to align the student and teacher models ); and wherein the second optimizer is configured to optimize reconstruction performance of the student VAE. (¶[0060], “In the cross distillation stage, the loss term consists of two parts: the reconstruction loss of urban landscape image supervision and the behavioral similarity loss based on CKA. The reconstruction loss consists of custom-character.sub.1 norm of super-resolution images and high-resolution city images output by two paths, namely, custom-character.sub.rec1 and custom-character.sub.rec2, which aims to make the output of the whole network as close to the real city image as possible.”, ¶[0060] disclose optimizing the reconstruction loss such that the output of the network is as close to the real city image as possible.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Lee with Hu in order to have a first optimizer to align the student and teacher model and a second to optimize reconstruction of the student. One skilled in the art would have been motivated to modify Lee in this manner in order for the performance of a student network can gradually approach the teacher network. (Hu, Abstract)
Regarding Claim 6, the combination of Lee and Hu teach the method of Claim 1, where Lee further teaches further comprising: after the training, deploying the student VAE for use with a generative artificial intelligence/machine learning (AI/ML) model. (Col 6, Lines 19-25, “The labeled input dataset 200 is used to train a classifier which may be a deep learning neural network such as, for example, a variational autoencoder (VAE) 202. Although a VAE is used as an example, a person of skill in the art should recognize that other deep generative models may be used instead of VAE, such as, for example, a generative adversarial network (GAN”, this section discloses using a generative AI model alongside a VAE.)
Regarding Claim 7, the combination of Lee and Hu teach the method of Claim 6, where Lee further teaches further comprising: using the student VAE and the AI/ML model to perform image-conditioned generation in order to generate output images based on input images. (Col 6, Lines 44-48, “the classifier engine 115 is also configured to create multiple (e.g. two or more) instances 208a-208c (collectively referred to as 208) of a binary baseline classifier. Each classifier instance 208 may be, for example, a neutral network.”, this section discloses generating multiple output images based on the input image.)
Regarding Claim 8, claim 8, is considered an apparatus claim substantially corresponding to claim 1. Please see the discussion of claim 1 above for a discussion of similar limitations. Furthermore, Lee teaches an apparatus comprising: at least one processing device configured to train a student variational autoencoder (VAE) based on a teacher VAE (see Fig 2 shows the architecture of the VAE and Figure 3 is a flowchart on the training process),
Regarding Claim 13, it is substantially similar to claim 6 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding Claim 14, it is substantially similar to claim 7 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding Claim 15, A method comprising: processing an input image using an encoder of a variational autoencoder (VAE) (Figure 3, Step 300 discloses processing the input image); providing an output of the encoder of the VAE to a generative artificial intelligence/ machine learning (AI/ML) model(Figure 3, Step 308 discloses inputting the image into a VAE); performing an image-conditioned generation task using the generative AI/ML model(Col 6, Lines 19-25 disclose generating multiple images using a generative AI.); and processing an output of the generative AI/ML model using a decoder of the VAE to generate an output image based on the input image(Col 6, Lines 44-48, “the classifier engine 115 is also configured to create multiple (e.g. two or more) instances 208a-208c (collectively referred to as 208) of a binary baseline classifier. Each classifier instance 208 may be, for example, a neutral network.”, this section discloses generating multiple output images based on the input image.); wherein training the student VAE based on the teacher VAE comprises using a first optimizer and a second optimizer(Col 8, Lines 30-34, “FIG. 4 is a more detailed block diagram of the VAE 202 according to one exemplary embodiment. In general terms, the VAE is a neural network that may include an encoder network (simply referred to as an “encoder”) 400 and a decoder network (simply referred to as a “decoder” 402.”, discloses the VAE is trained based on an encoder and decoder which the specification of the instant application discloses the first optimizer to be an encoder and the second optimizer is a decoder.)
Lee does not explicitly teach wherein the first optimizer is configured to align a latent space of the student VAE with a latent space of the teacher VAE;and wherein the second optimizer is configured to optimize reconstruction performance of the student VAE.
Hu teaches wherein the first optimizer is configured to align a latent space of the student VAE with a latent space of the teacher VAE ([0022], “Further, a purpose of knowledge transfer is achieved by aligning respective inter-layer relationship matrices of teachers and students; and a specific processing flow of output features X and Y of different layers is as follows”, ¶[0022]-¶[0024] disclose a process of using knowledge distillation to align the student and teacher models ); and wherein the second optimizer is configured to optimize reconstruction performance of the student VAE. (¶[0060], “In the cross distillation stage, the loss term consists of two parts: the reconstruction loss of urban landscape image supervision and the behavioral similarity loss based on CKA. The reconstruction loss consists of custom-character.sub.1 norm of super-resolution images and high-resolution city images output by two paths, namely, custom-character.sub.rec1 and custom-character.sub.rec2, which aims to make the output of the whole network as close to the real city image as possible.”, ¶[0060] disclose optimizing the reconstruction loss such that the output of the network is as close to the real city image as possible.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Lee with Hu in order to have a first optimizer to align the student and teacher model and a second to optimize reconstruction of the student. One skilled in the art would have been motivated to modify Lee in this manner in order for the performance of a student network can gradually approach the teacher network. (Hu, Abstract)
Regarding Claim 20, the combination of Lee and Hu teach The method of Claim 15, where Hu further teaches wherein the image-conditioned generation task comprises at least one of: image restoration and low-light denoising. (¶0006] discloses image restoration for the input image.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Lee with Hu in order to perform image restoration on the input image. One skilled in the art would have been motivated to modify Lee in this manner in order for the performance of a student network can gradually approach the teacher network. (Hu, Abstract)
Claims 2, 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. US Patent (US 11710045 B2) in view of Hu et al. US PG-Pub(US 20240233077 A1) in view of Rao et al. ("Parameter-Efficient and Student-Friendly Knowledge Distillation").
Regarding Claim 2, while Lee and Hu teach the method of Claim 1, they do not explicitly teach wherein, during use of the first optimizer while training the student VAE: parameters of the teacher VAE are frozen and parameters of an encoder of the student VAE are adjusted using the first optimizer to minimize a loss between outputs of an encoder of the teacher VAE and outputs of the encoder of the student VAE.
Rao teaches wherein, during use of the first optimizer while training the student VAE: parameters of the teacher VAE are frozen; and parameters of an encoder of the student VAE are adjusted using the first optimizer to minimize a loss between outputs of an encoder of the teacher VAE and outputs of the encoder of the student VAE. (Page 4233, Knowledge Distillation with Adapter (PESF-KD), Training Objective, “Our approach achieves better performance and less training time compared to DML[17]by introducing the adapter module and adjusting soft labels to provide the smoothed soft labels obtained by network training that are more reason able compared to LS and temperature adjustment, respectively. Formally, the training loss of the student and teacher network can be formulated as follows: Ls =αLKL p(θs),p(θta) +(1−α)Ltask(θs), (7) Lt =αLKL p(θta)p(θs) +(1−α)Ltask(θta), (8) where θta is the parameter of the adapter of the teacher network needed to update.”, this section disclose the loss of the student and teacher are minimized by adjusting the parameters of the student.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Lee and Hu with Rao in order to freeze the teacher parameters during training. One skilled in the art would have been motivated to modify Lee and Hu in this manner in order to transfer knowledge using the soft label from this pre-trained teacher model to a smaller student. (Rao, Abstract)
Regarding Claim 9, it is substantially similar to claim 2 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding Claim 16, it is substantially similar to claim 2 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 3-4, 10-11 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. US Patent (US 11710045 B2) in view of Hu et al. US PG-Pub(US 20240233077 A1) in view of Rao et al. ("Parameter-Efficient and Student-Friendly Knowledge Distillation") in view of Ye et al. ("Continual Variational Autoencoder via Continual Generative Knowledge Distillation").
Regarding Claim 3, while the combination of Lee, Hu and Rao teach the method of Claim 2, they do not explicitly teach wherein, during use of the second optimizer while training the student VAE: the parameters of the encoder of the student VAE are frozen; and parameters of a decoder of the student VAE are adjusted using the second optimizer to minimize a loss between inputs to the encoder of the student VAE and outputs of the decoder of the student VAE.
Ye teaches wherein, during use of the second optimizer while training the student VAE: the parameters of the encoder of the student VAE are frozen(Page 109120, Right Col, First paragraph “If Eq. (1) is satisfied, we freeze Ac that has already preserved the knowledge of the current memory Mi, while adding a new expert Ac+1 for next training step” discloses freezing the student and adding a new teacher for the next training step.); and parameters of a decoder of the student VAE are adjusted using the second optimizer to minimize a loss between inputs to the encoder of the student VAE and outputs of the decoder of the student VAE(Page 10920, Continual Generative Knowledge Distillation, Paragraph 1-3 disclose the student VAE is adjusted to reduce the loss between the encoder and decoder.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Lee, Hu and Rao with Ye in order to freeze the student parameter during training. One skilled in the art would have been motivated to modify Lee, Hu and Rao in this manner in order to develop a new unsupervised continual learning framework consisting of two memory systems using Variational Autoencoders (VAEs). (Ye, Abstract)
Regarding Claim 4, the combination of Lee, Hu, Rao and Ye teach the method of Claim 3, where Hu further teaches wherein: the first optimizer is configured to minimize an L1 loss between the latent space of the student VAE and the latent space of the teacher VAE; and the second optimizer is configured to minimize a combination of an L1 loss and a divergence loss between the inputs to the encoder of the student VAE and the outputs of the decoder of the student VAE. (¶[0020], “calculating a loss term based on the preliminary image result and the high-definition image to punish an incorrect image restoration by the network, where the loss term includes whether the structure is consistent, whether color conforms to a statistical law and whether the texture is natural, and finally, optimizing parameters of the two paths based on the loss; and [0021] (3) taking out and connecting two cross-distilled student modules to form a final optimized path; inputting the low-definition image into the network; generating a super-resolution image for supervision using a complete teacher network, to calculate the loss term and update the parameters”, ¶[0020]-¶[0021] discloses minimize the loss between the student and teacher alongside reducing the loss in the student mode.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Lee, Rao and Ye with Hu in order to minimize the loss of the student teacher networks. One skilled in the art would have been motivated to modify Lee, Rao and Ye in this manner in order for the performance of a student network can gradually approach the teacher network. (Hu, Abstract)
Regarding Claim 10, it is substantially similar to claim 3 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding Claim 11, it is substantially similar to claim 4 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding Claim 17, it is substantially similar to claim 3 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding Claim 18, it is substantially similar to claim 4 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 5, 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. US Patent (US 11710045 B2) in view of Hu et al. US PG-Pub(US 20240233077 A1) in view of Qi et al. ("Multi-Scale Aligned Distillation for Low-Resolution Detection").
Regarding Claim 5, while the combination of Lee and Hu teach the method of Claim 1, further comprising: up-sampling inputs to the teacher VAE during the training of the student VAE (Page 5, 3.3 Cross Feature-Level Knowledge Distillation, Paragraph 1, “With aligned multi-scale training and crossing feature level fusion, we obtain a strong multi-scale fusion teacher whose multi-resolution features can be seamlessly distilled to the low-resolution student. Similar to those in previous sections, we denote high resolution and input resolution as H ×W and H×W used by the teacher and student respectively”, in this section of the prior art the teacher’s input are at a higher resolution than the student’s.); wherein the student VAE has a common design as the teacher VAE but lacks at least one down-sampling operation that is included in the teacher VAE (Fig. 2 shows the student-teacher having similar designs however the teacher has a low and high resolution scaling.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Lee and Hu with Qi in order to have the teacher network be at a higher scale than the student. One skilled in the art would have been motivated to modify Lee and Hu in this manner in order to dynamically fuse teacher’s multi-resolution features to guide the student better. (Qi, Abstract).
Regarding Claim 12, it is substantially similar to claim 5 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding Claim 19, it is substantially similar to claim 5 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAN D HOANG whose telephone number is (571)272-4344. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN M VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAN HOANG/Primary Examiner, Art Unit 2661
Prosecution Insights