Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Figures 1-2, 4-7 should be designated by a legend such as --Prior Art-- because only that which is old is illustrated. See MPEP § 608.02(g). Corrected drawings in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. The replacement sheet(s) should be labeled “Replacement Sheet” in the page header (as per 37 CFR 1.84(c)) so as not to obstruct any portion of the drawing figures. If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The disclosure is objected to because of the following informalities: they fail to show structural detail of Figure 7.
Appropriate correction is required.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over ZHANG et al (Adding Conditional Control to Text-to-Image Diffusion Models) in view of TUMANYAN et al (Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation).
As per claim 1, Zhang teaches the claimed “computer-implemented method of generating synthetic images using a conditional diffusion model,” the method comprising the following steps: “providing a neural conditioning for a ControlNet as input, wherein the neural conditioning has been determined by a foundation model for the to be generated synthetic image” (Zhang, 3.2. ControlNet for Text-to-Image Diffusion - The ControlNet structure is applied to each encoder level of the U-net (Figure 3b). In particular, we use ControlNet to create a trainable copy of the 12 encoding blocks and 1middle block of Stable Diffusion. The 12 encoding blocks are in 4 resolutions (64 × 64, 32 × 32, 16 × 16, 8 × 8) with each one replicated 3 times. The outputs are added to the 12 skip-connections and 1 middle block of the U-net. Since Stable Diffusion is a typical U-net structure, this ControlNet architecture is likely to be applicable with other models… To add ControlNet to Stable Diffusion, we first convert each input conditioning image (e.g., edge, pose, depth, etc.) from an input size of 512 × 512 into a 64 × 64 feature space vector that matches the size of Stable Diffusion. In particular, we use a tiny network E(·) of four convolution layers with 4 × 4 kernels and 2 × 2 strides (activated by ReLU, using 16, 32, 64, 128, channels respectively, initialized with Gaussian weights and trained jointly with the full model) to encode an image-space condition ci into a feature space conditioning vector Cf as, Cf = E(Ci). The conditioning vector Cf is passed into the ControlNet) (Noted: Zhang’s input to the ControlNet Cf can be the neural condition determined by a foundation model for the to be generated synthetic image as showed by Tumanyan (e.g., 4. Method - Our key finding is that fine-grained control over the generated structure can be achieved by manipulating spatial features inside the model during the generation process. Specifically, we observe and empirically demonstrate that: (i) spatial features extracted from intermediate decoder layers encode localized semantic information and are less affected by appearance information, and (ii) the self-attention, representing the affinities between the spatial features, allows to retain fine layout and shape details…); “propagating the neural conditioning and a latent input representation for the diffusion model through the ControlNet, and providing outputs of the ControlNet as additional injections for the diffusion model” (Zhang, 3.2. ControlNet for Text-to-Image Diffusion - The way we connect the ControlNet is computationally efficient — since the locked copy parameters are frozen, no gradient computation is required in the originally locked encoder for the finetuning. This approach speeds up training and saves GPU memory); and “propagating the latent input representation through the diffusion model, wherein during the propagating of the latent input representation, the additional injections from the ControlNet are injected into corresponding layers of the diffusion model” (Zhang, Figure 3: Stable Diffusion’s U-net architecture connected with a ControlNet on the encoder blocks and middle block. The locked, gray blocks show the structure of Stable Diffusion V1.5 (or V2.1, as they use the same U-net architecture). The trainable blue blocks and the white zero convolution layers are added to build a ControlNet; Tumanyan, 4. Method, Spatial features - As seen, the coarsest and shallowest layer is mostly dominated by foreground-background separation, depicting only a crude blob in the location of the foreground object. Interestingly, we can observe that the intermediate features (layer 4) encode localized semantic information shared across objects from different domains and under significant appearance variations – similar object parts (e.g., legs, torso, head) are depicted in similar colors across all images (layer=4 row in Fig. 3). These proper ties are consistent across the generation process as shown in Fig. 4; Feature Injection - Fig. 5(a) shows the effect of injecting spatial features flt at increasing layers l. As seen, injecting features only at layer l=4 is insufficient for preserving the structure of the guidance image. As we inject features in deeper layers, the structure is better preserved, yet appearance information is leaked into the generated image (e.g., shades of the red t-shirt and blue jeans are apparent in Layer 4-11)). Thus, it would have been obvious, in view of Tumanyan, to configure Zhang’s method as claimed by using the neural conditioning as input for a ControlNet and propagating the neural conditioning and a latent input representation for the diffusion model through the ControlNet. The motivation is to achieve the control of fine-grained over the generated structure by manipulating spatial features inside the model.
Claim 2 adds into claim 1 “wherein the neural conditioning is determined by propagating the to be generated synthetic image through the foundation model and selecting a plurality of intermediate results of the foundation model as the neural conditioning” (Tumanyan, 4. Method, Spatial features - As seen, the coarsest and shallowest layer is mostly dominated by foreground-background separation, depicting only a crude blob in the location of the foreground object. Interestingly, we can observe that the intermediate features (layer 4) encode localized semantic information shared across objects from different domains and under significant appearance variations – similar object parts (e.g., legs, torso, head) are depicted in similar colors across all images (layer=4 row in Fig. 3). These proper ties are consistent across the generation process as shown in Fig. 4; Feature Injection - Fig. 5(a) shows the effect of injecting spatial features flt at increasing layers l. As seen, injecting features only at layer l=4 is insufficient for preserving the structure of the guidance image. As we inject features in deeper layers, the structure is better preserved, yet appearance information is leaked into the generated image (e.g., shades of the red t-shirt and blue jeans are apparent in Layer 4-11)). Thus, it would have been obvious, in view of Tumanyan, to configure Zhang’s method as claimed by determining the neural conditioning using propagating the to be generated synthetic image through the foundation model and selecting a plurality of intermediate results of the foundation model. The motivation is to achieve the control of fine-grained over the generated structure by manipulating spatial features inside the model.
Claim 3 adds into claim 2 “wherein a Principal Component Analysis or a machine learning system is applied to the plurality of intermediate results to obtain the neural conditioning” (Tumanyan, 4. Method, Spatial features - We opt to gain a better understanding of how such semantic spatial information is internally encoded in Ɛθ. To this end, we perform a simple PCA analysis which allows us to reason about the visual properties dominating the high dimensional features in Ɛθ. Specifically, we generated a diverse set of images containing various humanoids in different styles, including both real and text-generated images; sample images are shown in Fig. 3. For each image, we extract features flt from each layer of the decoder at each time step t, as illustrated in Fig. 2(b). We then apply PCA on flt across all images). Thus, it would have been obvious, in view of Tumanyan, to configure Zhang’s method as claimed by applying a Principal Component Analysis, or a machine learning system to the plurality of intermediate results to obtain the neural conditioning. The motivation is to achieve the control of fine-grained over the generated structure by manipulating spatial features inside the model.
Claim 4 adds into claim 1 “wherein the neural conditioning is a per-pixel neural representation of a reference image” (Tumanyan, 4. Method - Based on our findings, we devise a simple framework that extracts features from the generation process of the guidance image IG and directly injects them along with P into the generation process of I∗, requiring no training or fine-tuning (Fig. 2). Our approach is applicable for both text-generated and real-world guidance images, for which we apply DDIM inversion to get the initial xGT). Thus, it would have been obvious, in view of Tumanyan, to configure Zhang’s method as claimed by collecting the neural conditioning as per-pixel neural representation of a reference image. The motivation is to achieve the control of fine-grained over the generated structure by manipulating spatial features inside the model.
Claim 5 adds into claim 1 “wherein the diffusion model includes a forward diffusion process and a backward denoising process” (Tumanyan, 3. Preliminary - These diffusion models are founded on two complementary random processes, the forward process, in which Gaussian noise is progressively added to a clean image x0 …, and the backward process is aimed at gradually denoising xT, where at each step a cleaner image is obtained), “wherein for training the diffusion model, the following steps are performed: obtaining a given image, encoding the given image into a latent code using an encoder of an autoencoder, generating a noisy latent code by adding Gaussian noise to clean latent code according to a fixed variance schedule, and decoding the latent code back to the image space using a decoder of the autoencoder” (Tumanyan, 3. Preliminary - Diffusion models are probabilistic generative models in which an image is generated by progressively removing noise from an initial Gaussian noise image, xT∼N(0, I). These diffusion models are founded on two complementary random processes, the forward process, in which Gaussian noise is progressively added to a clean image x0 …, and the backward process is aimed at gradually denoising xT, where at each step a cleaner image is obtained… In this work, we leverage a pre-trained text-conditioned Latent Diffusion Model (LDM), a.k.a Stable Diffusion, in which the diffusion process is applied in the latent space of a pre-trained image autoencoder).
Claims 8 and 9 claim a non-transitory machine-readable storage medium and a system based on the method of claims 1-5; therefore, they are rejected under a similar rationale.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over ZHANG et al (Adding Conditional Control to Text-to-Image Diffusion Models) in view of TUMANYAN et al (Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation).
Claim 6 adds into claim 1, Zhang does not teach, but Zeme teaches “wherein a synthetic image generated using the conditional diffusion model is used for training an image classifier” (Zeme, page 1 - A machine-learning model for image classification that’s trained using synthetic data can rival one trained on the real thing, a study shows; page 2, 2nd paragraph - MIT researchers developed a method for training a machine learning model that, rather than using a dataset, uses a special type of machine-learning model to generate extremely realistic synthetic data that can train another model for downstream vision tasks). Thus, it would have been obvious, in view of Tumanyan and Zeme, to configure Zhang’s method as claimed by using a synthetic image generated by the conditional diffusion model for training an image classifier. The motivation is to reduce required memory to share or store the real data on a database, or to be able to edit for removing certain attributes, like race or gender, which could address some biases existed in traditional real datasets.
Claim 7 adds into claim 6 “wherein the image classifier is used for controlling an at least partially autonomous robot and/or a manufacturing machine and/or an access control system” (Zeme, page 2, 2nd paragraph - MIT researchers developed a method for training a machine learning model that, rather than using a dataset, uses a special type of machine-learning model to generate extremely realistic synthetic data that can train another model for downstream vision tasks…; page 3, 4th paragraph - If the model is trained on images of cars, it can “imagine” how a car would look in different situations — situations it did not see during training — and then output images that show the car in unique poses, colors, or sizes; page 4, 1st paragraph - For instance, if researchers are training a computer vision model for a self-driving car, real data wouldn’t contain examples of a dog and his owner running down a highway, so the model would never learn what to do in this situation. Generating that corner case data synthetically could improve the performance of machine learning models in some high-stakes situations). Thus, it would have been obvious, in view of Tumanyan and Zeme, to configure Zhang’s method as claimed by using the image classifier for controlling an at least partially autonomous robot and/or a manufacturing machine and/or an access control system. The motivation is to reduce required memory to share or store the real data on a database, or to be able to edit for removing certain attributes, like race or gender, which could address some biases existed in traditional real datasets.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/Primary Examiner, Art Unit 2616
Prosecution Insights