DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
STATUS OF CLAIMS
Claims 1-26 are pending
Claims 8, 14, 16, 17, 19, and 20 are canceled
Claims 1, 3, 10, and 15 are amended
Claims 21-26 are new
Claim Rejections - 35 USC § 112
Claim 15, 18, and 21-26 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention, “modify an encoded image to include (i) first noise values within a first region of the encoded image and (ii) second noise values within a second region of the encoded image, wherein the first noise values and the second noise values represent different statistical distributions of randomized noise; generating, using (i) the modified encoded image and (ii) the SD model trained using output of a second model, different than the SD model, and a plurality of training images representing plain colored backgrounds each paired with a text prompt indicating the corresponding plain background, a latent representation of the modified encoded image; and generating, using the generated latent representation and the decoder,” contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7, 8, 15, and 16 are rejected under 35 U.S.C. 103 as being anticipated by Burgert in view of Wang et al. (CN 116824004) and Guo et al. (CN 116630464) et al.
Regarding claim 1:
Burgert, Guo, and Wang together teach:
An apparatus comprising:
at least one processor assembly configured to:
modify a stable diffusion (SD) model to generate (Burgert [Pg 1 Par 3] Our proposed Peekaboo, based on a pre-trained image-language stable diffusion model [17],)(Guo [Abstract] the first model is a LoRA model obtained by finely adjusting the cross attention layer in the UNet module in the stable diffusion model;)(Wang [Page 3 Paragraph 9] In addition, icons can also be generated by means of AIGenerated Content (AIGC), for example, based on stable diffusion),
Burgert teaches:
based on modifying a noise distribution component of the SD model (Burgert [Pg 4 Par 2] First introduced in DreamFusion [45], Score Distillation Sampling (SDS) is a method that generates samples from a diffusion model by optimizing a loss function we call score distillation loss (SDL). This allows us to optimize samples in any parameter space, as long as we can map back to images in a differentiable manner. We modify SDS to optimize learnable alpha masks and operate with latent diffusion. [Pg 4 Par 7 The Stable Diffusion model jointly processes images and text. Its visual encoder first projects images
to a latent space. This latent vector is then processed by the diffusion U-Net (D) conditioned on text embedding (from text encoder T ) to produce noise outputs. To measure Ls, we first degrade latent vector z of composite image x using forward diffusion, introducing Gaussian noise ϵ ∼ N to z, resulting in a noisy ˜z. We then perform diffusion denoising conditioned on the text embedding with pre-trained D. Our loss Ls is measured as the reconstruction error of noise ϵ, given noisy ˜z and text embedding T (p) as in Eq. (2): Ls = MSE(ϵ,D(˜z,T (p))) (2)
where MSE refers to mean-squared loss. Pseudo-code describing Ls in detail is presented in Algorithm 1)
Gou teaches:
and using a decoder of the SD model that is trained separately from a corresponding encoder of the SD model that forms a variational autoencoder (VAE) with the decoder of the SD model (Gou [ABSTRACT] An image generation network is constructed (S102) based on a stable diffusion model. An image encoder and an image decoder are trained (S103) by a low-rank adaptation method in combination with original image samples and image description text in the pre-training data set to obtain a positive sample generation model. A controlled image generation network is constructed (S104). The image encoder and the image decoder are trained (S105) by using a low-rank adaptation method to obtain a negative sample generation model)
Burgert and Wang teach:
from a first text prompt, a first image having red, green, blue, and alpha (RGBA) channels (Burgert [Pg 4 Par 1] We formulate segmentation as a foreground alpha mask optimization problem and leverage a text-to-image stable diffusion model pre-trained on large internet-scale data. The alpha mask is optimized with respect to image and text prompts.)(Wang [Pg 7 Par 3] The icons of the four channels may include red, green, blue and transparency four channels, wherein the transparency (Alpha) channel stores the transparency value of each pixel for controlling the visibility of the pixels in the image.);
and responsive to the first text prompt, output the first image (Wang [Pg4 Par 2] Finally, the generated icon is sent to the terminal device 101 through the network for display, so as to perform subsequent processing based on the icon.) (Burgert [Pg 8 Par 3] Peekaboo loss can do more than just segment images - it can generate images with transparency. In fact, it does this quite reliably - giving detailed alpha masks along with each generated image. Images with transparency are incredibly useful for many domains, such as graphic design assets and video game textures. However, to the best of our knowledge, all current text-to-image models such as [38][77][40][37] have trained strictly on RGB images, and are not designed to generate images with an alpha channel. Despite the absence of such models, we can generate RGBA images by iteratively optimizing an RGB image jointly with an alpha mask using Peekaboo loss. Pseudocode for this process is given in Algorithm 2, which builds on Algorithm 1).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Guo and Wang. Having transparency and being able to pick colors, as in Guo and Wang, would benefit the Burgert teachings by having a way to have transparent backgrounds on images. Additionally, this is the application of a known technique, Having transparency and being able to pick colors, to yield predictable results.
Regarding claim 2:
Burgert, Guo, and Wang teach:
The apparatus of Claim 1,
wherein the alpha channel represents a transparent or solid-colored background around an image of an object (Wang [Pg 7 Par 3] The icons of the four channels may include red, green, blue and transparency four channels, wherein the transparency (Alpha) channel stores the transparency value of each pixel for controlling the visibility of the pixels in the image.).
Regarding claim 3:
Burgert, Guo, and Wang teach:
The apparatus of Claim 1,
wherein the processor assembly is configured to: modify a noise distribution of the SD model to enable the SD model to output centered objects with low-variance backgrounds (Wang [Pg 6 Par 4] The foreground mask image is a single-channel image, which can represent the position information of the interested foreground object in the image, wherein the area where the foreground object is located can be black, and the background area can be white.);
tune a U-Net of the SD model to allow the SD model to recognize the noise distribution modified to output centered objects with low-variance backgrounds (Wang [Pg 6 Par 5] Specifically, the matting model can be trained by using the U2-Net network, wherein the U2-Net is a neural network based on deep learning and can be used for image segmentation and matting task of pixel level.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Guo and Wang. Having transparency and being able to pick colors, as in Guo and Wang, would benefit the Burgert teachings by having a way to have transparent backgrounds on images. Additionally, this is the application of a known technique, Having transparency and being able to pick colors, to yield predictable results.
Regarding claim 4:
Burgert, Guo, and Wang teach:
The apparatus of Claim 3,
wherein the processor assembly is configured to: train a decoder of the SD model to output RGBA images using the U-Net (Wang [Pg 9 Par 9] The decoder can decode (i.e., decompress) the coded data of the image or video to restore the image or video data. [Pg 6 Par 5] In the embodiment of the disclosure, the matting model can be trained in advance before matting the composite image. Specifically, the matting model can be trained by using the U2-Net network).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Guo and Wang. Having transparency and being able to pick colors, as in Guo and Wang, would benefit the Burgert teachings by having a way to have transparent backgrounds on images. Additionally, this is the application of a known technique, Having transparency and being able to pick colors, to yield predictable results.
Regarding claim 7:
Burgert, Guo, and Wang teach:
The apparatus of Claim 3,
wherein the processor assembly is configured to tune the U-net at least in part by:
executing a tuning method on plural images with plain white backgrounds and respective corresponding text prompts followed by keywords “no background” using the noise distribution to train the SD model to produce centered foreground images with plain-colored backgrounds (Wang [Pg 4 Par 11] In the embodiment of the present disclosure, the entry may be a piece of text description or instruction for directing the model to generate particular image content. The terms may be used to characterize features of an image to be generated, for example, may be used to represent any image-related attributes of the type, action, scene, colour, and the like of the image to be generated. The description is carried out by the prompt word so as to generate the corresponding image.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Guo and Wang. Having transparency and being able to pick colors, as in Guo and Wang, would benefit the Burgert teachings by having a way to have transparent backgrounds on images. Additionally, this is the application of a known technique, Having transparency and being able to pick colors, to yield predictable results.
Regarding claim 8:
Burgert, Guo, and Wang teach:
The apparatus of Claim 7,
wherein the tuning method comprises Low Rank Adaptation (LoRA) (Wang [Pg 5 Par 8] The sample icons in the same style and the corresponding terms form sample data, and the low-rank adaptation model (LoRA) is trained by the sample data to obtain an icon generation model. The LoRA model can be regarded as a model of local finetune of diffusion model, the storage is small, and the LoRA model can be customized, and the characteristics of the appointed object data can be obtained by learning. training the LoRA model by using the sample data composed of the sample icon and the abstract, finally obtaining the icon generation model. Exemplary, the LoRA model may be trained based on a typer-discharge v1.5.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Guo and Wang. Having transparency and being able to pick colors, as in Guo and Wang, would benefit the Burgert teachings by having a way to have transparent backgrounds on images. Additionally, this is the application of a known technique, Having transparency and being able to pick colors, to yield predictable results.
Regarding claim 15:
Burgert, Guo, and Wang teach:
An apparatus comprising: at least one computer medium that is not a transitory signal and that comprises instructions executable by at least one processor assembly to: modify at least one decoder of at least one stable diffusion (SD) model to generate images having red, green, and blue (RGB) data and data indicating transparency (Burgert [Pg 1 Par 3] Our proposed Peekaboo, based on a pre-trained image-language stable diffusion model [17],)(Guo [Abstract] the first model is a LoRA model obtained by finely adjusting the cross attention layer in the UNet module in the stable diffusion model;)(Wang [Page 3 Paragraph 9] In addition, icons can also be generated by means of AIGenerated Content (AIGC), for example, based on stable diffusion) (Guo [Pg 3 Par 2] building a stable diffusion model, performing condition training to the UNet encoder and UNet decoder of the UNet module in the stable diffusion model, so as to obtain the second model; The conditional training is training with constraint conditions. Further: processing the input image using the first model to obtain a first image, comprising the following steps:)(Wang [Page 3 Paragraph 9] In addition, icons can also be generated by means of AIGenerated Content (AIGC), for example, based on stable diffusion) (Wang [Pg 7 Par 3] The icons of the four channels may include red, green, blue and transparency four channels, wherein the transparency (Alpha) channel stores the transparency value of each pixel for controlling the visibility of the pixels in the image.) (Wang [Pg 9 Par 9] The decoder can decode (i.e., decompress) the coded data of the image or video to restore the image or video data. [Pg 6 Par 5] In the embodiment of the disclosure, the matting model can be trained in advance before matting the composite image. Specifically, the matting model can be trained by using the U2-Net network) (Burgert [Pg 4 Par 1] We formulate segmentation as a foreground alpha mask optimization problem and leverage a text-to-image stable diffusion model pre-trained on large internet-scale data. The alpha mask is optimized with respect to image and text prompts.)(Wang [Pg 7 Par 3] The icons of the four channels may include red, green, blue and transparency four channels, wherein the transparency (Alpha) channel stores the transparency value of each pixel for controlling the visibility of the pixels in the image.);
and responsive to a text prompt input to the SD model, receive from the SD model at least one image having RGB data and data indicating transparency (Wang [Pg 4 Par 11] In the embodiment of the present disclosure, the entry may be a piece of text description or instruction for directing the model to generate particular image content. The terms may be used to characterize features of an image to be generated, for example, may be used to represent any image-related attributes of the type, action, scene, colour, and the like of the image to be generated. The description is carried out by the prompt word so as to generate the corresponding image.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Guo and Wang. Having transparency and being able to pick colors, as in Guo and Wang, would benefit the Burgert teachings by having a way to have transparent backgrounds on images. Additionally, this is the application of a known technique, Having transparency and being able to pick colors, to yield predictable results.
Regarding claim 16:
Burgert, Guo, and wang teach:
The apparatus of Claim 15,
wherein the data indicating transparency indicates a transparent background around an image of an object (Wang [Pg 7 Par 1] The composite image is a three-channel image, and the foreground mask image is a single-channel image, the single-channel foreground mask image can be used as a fourth channel of the composite image, namely a transparency channel, so that the finally generated icon background is consistent and transparent.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Guo and Wang. Having transparency and being able to pick colors, as in Guo and Wang, would benefit the Burgert teachings by having a way to have transparent backgrounds on images. Additionally, this is the application of a known technique, Having transparency and being able to pick colors, to yield predictable results.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), and Dong et al. (CN 103186888).
Regarding claim 5:
Burgert, Guo, and Wang teach:
The apparatus of Claim 3,
Wang fails to teach:
wherein the processor assembly is configured to modify the noise distribution at least in part by: establishing a first noise profile in an inner circle of latents;
and establishing a second noise profile outside the inner circle.
Dong teaches:
wherein the processor assembly is configured to modify the noise distribution at least in part by: establishing a first noise profile in an inner circle of latents (Dong [Claim 3] calculating in the outer envelope of the image respectively based on the morphological open operation and closed operation to the image pixel value after weighting. respectively obtaining the inner envelope of the image and enveloping, f) calculating the noise profile of the inner and outer envelope calculating average value to obtain the smooth distribution SmoothMap and calculating the noise map NoiseMap = abs (EstimationMap-SmoothMap););
and establishing a second noise profile outside the inner circle (Dong [Claim 3] calculating in the outer envelope of the image respectively based on the morphological open operation and closed operation to the image pixel value after weighting. respectively obtaining the inner envelope of the image and enveloping, f) calculating the noise profile of the inner and outer envelope calculating average value to obtain the smooth distribution SmoothMap and calculating the noise map NoiseMap = abs (EstimationMap-SmoothMap);).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang with Dong. Establishing noise profiles, as in Dong, would benefit the Wang teachings by having profiles for the noise data. Additionally, this is the application of a known technique, creating noise profiles for the noise distribution, to yield predictable results.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), Dong et al. (CN 103186888) and Nguyen et al (EP 3340214).
Regarding claim 6:
Burgert, Guo, Wang, and Dong teach:
The apparatus of Claim 5,
Wang and Dong fail to teach:
wherein the first noise profile is uniformly random noise and the second noise profile is offset noise that enables the SD model to learn to change a zero-frequency of the component.
Nguyen teaches:
wherein the first noise profile is uniformly random noise and the second noise profile is offset noise that enables the SD model to learn to change a zero-frequency of the component (Nguyen [Pg 6 Par 4] These changes are possible in the measuring instrument or environmental condition occurs. "noise" refers to random deviation of the signal changes with time. noise is random error such that it can be obtained by signal processing (e.g., filtering) to reduce, typically at the cost of dynamic behaviour of the sensor. noise or noise profile may be characterized, and further by using the present invention. [Pg 13 Par 7] For example, in order to determine the noise profile (distribution) within a few seconds the computing device of the sensor embedded in the considered set in static state is sufficient (in such a case, determining the offset variation of the accelerometer).).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang and Dong with Nguyen. Having random and a non-random noise, as in Nguyen, would benefit the Wang and Dong teachings by having different types of noise data. Additionally, this is the application of a known technique, having multiple different types of noise, to yield predictable results.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), and Xu et al. (US 20210319564).
Regarding claim 9:
Burgert, Guo, and Wang teach:
The apparatus of Claim 4,
Wang fails to teach:
wherein the processor assembly is configured to train the decoder to output RGBA images at least in part by: training the decoder to predict the alpha channel from an image with a plain-colored, low variance background output by the U-net.
Xu teaches:
wherein the processor assembly is configured to train the decoder to output RGBA images at least in part by: training the decoder to predict the alpha channel from an image with a plain-colored, low variance background output by the U-net (Xu [0078] Decoder 528 predicts alpha values for each pixel within query image patch 506 and may be similar to decoder 326 in FIG. 3. Alpha values for all pixels within image patch 506 are combined to generate a matte patch 530 that is output from the neural network system 520.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang with Xu. Training a decoder with the alpha channel, as in Xu, would benefit the Wang teachings by having background data for the alpha channel. Additionally, this is the application of a known technique, training a decoder based on the alpha chanel, to yield predictable results.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), Xu et al. (US 20210319564), and Friedman et al. (US 20210019603).
Regarding claim 10:
Burgert, Guo, Wang and Xu teach:
The apparatus of Claim 9,
Wang and Xu fail to teach:
wherein the processor assembly is configured to train the decoder to predict the alpha channel at least in part by: modifying a variational autoencoder (VAE) portion of the decoder to output a fourth channel encoding alpha information;
not training an encoder of the SD model during training of the decoder, so that only the decoder is modified and a learned latent distribution remains unchanged such that the SD model predicts the fourth channel of an image based on a latent representation of the image.
Friedman teaches:
wherein the processor assembly is configured to train the decoder to predict the alpha channel at least in part by: modifying a variational autoencoder (VAE) portion of the decoder to output a fourth channel encoding alpha information (Friedman [0037] In some examples, the encoder neural network 305 and the decoder neural network 310 are trained together using a variational autoencoder [0037] In some examples, the decoder neural network 310 generates α, β parameters as the probability model parameters 315 for a Binomial distribution to generate ratings.);
not training an encoder of the SD model during training of the decoder, so that only the decoder is modified and a learned latent distribution remains unchanged such that the SD model predicts the fourth channel of an image based on a latent representation of the image (Friedman [0065] Example 5 includes the apparatus of example 1, wherein the training controller is to train the neural network at least in part with a variational autoencoder, the variational autoencoder including an encoder and a decoder.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang and Xu with Friedman. Modifying the autoencoder, as in Friedman, would benefit the Wang and Xu teachings by allowing for a custom autoencoder. Additionally, this is the application of a known technique, modifying an autoencoder, to yield predictable results.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), Xu et al. (US 20210319564), and He et al. (WO 2022271146).
Regarding claim 11:
Burgert, Guo, Wang and Xu teach:
The apparatus of Claim 9,
Wang and Xu fail to teach:
wherein the processor assembly is configured to train the decoder to output RGBA images using a dataset comprising plural RGBA images with transparent backgrounds and versions of the plural images generated by applying one or more of random flips, rotations, zooms, and color augmentations of the plural images.
He teaches:
wherein the processor assembly is configured to train the decoder to output RGBA images using a dataset comprising plural RGBA images with transparent backgrounds and versions of the plural images generated by applying one or more of random flips, rotations, zooms, and color augmentations of the plural images (He [0060] In order to enhance the generalization potential of the decoder machine learning model 134a can augment the second dataset by generating new training samples using the existing training samples of the second dataset. To generate the new training samples, the training process can distort training images among the set of training images to create distorted images that are used to train the model. In some implementations, the distorted images can be generated by applying visual perturbations that widely occur in real-world visual data such as horizontal and vertical flips, translations, rotation, cropping, color distortions, adding random noise etc. In some implementations, the training process can generate new training samples by encoding the training images into different file formats using lossy compression or transformation techniques. [0043] For example, a 32-bit RGB pixel includes 8 bits for each color channel (e.g., Red (R), Green (G) and Blue (B)) and an “alpha” channel for transparency.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang and Xu with He. Having images that apply an effect to them, as in He, would benefit the Wang and Xu teachings by allowing for a wider set of images that may be used. Additionally, this is the application of a known technique, applying effects to images, to yield predictable results.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), Xu et al. (US 20210319564), He et al. (WO 2022271146), and Kim et al (KR 20160059488).
Regarding claim 12:
Wang, Xu, and He teach:
The apparatus of Claim 11,
Wang, Xu, and He fail to teach:
Kim teaches:
wherein the processor assembly is configured to: transform at least some of the RGBA images in the dataset into respective RGB images by replacing background in the some of the RGBA images with randomly generated, low-variance backgrounds (Kim [Pg 4 Par 3] The image conversion engine 325 replaces the transparent color of the transparent image with the background color or the background image and generates an opaque image and then performs an encoding operation with a high compression ratio opaque image storage method and outputs the result to the relevant temporary storage device 232.);
Dong teaches:
input the respective RGB images into the decoder to cause the decoder to predict corresponding RGBA image (Dong [Pg 15 Par 8] the color image will have RGBA four channels, RGB respectively red, green; blue three basic color channels, A represents the Alpha channel, describing the transparency of the image, but in the real process, in the image shooting process, it will not record the Alpha information, therefore, the instance segmentation module predicts the Alpha information of the image.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang, Xu, and He with Kim and Dong. Replacing the background in images, as in Kim and Dong, would benefit the Wang, Xu, and He teachings by allowing for a way to change the background in an image. Additionally, this is the application of a known technique, changing the background of an image, to yield predictable results.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (CN 116824004) in view of Xu et al. (US 20210319564), He et al. (WO 2022271146), Kim et al (KR 20160059488), and Ozcan et al (US 20210043331).
Regarding claim 13:
Wang, Xu, and He teach:
The apparatus of Claim 11,
Wang, Xu, and He fail to teach:
wherein the processor assembly is configured to: replace transparent pixels in the at least some of the RGBA images with a random-colored background image to convert the at least some of the RGBA images back to respective images in RGB space;
and determine mean squared error (MSE) loss for the respective images in RGB space such that SD model weights only visible pixels as important for alpha channel prediction.
Kim teaches:
wherein the processor assembly is configured to: replace transparent pixels in the at least some of the RGBA images with a random-colored background image to convert the at least some of the RGBA images back to respective images in RGB space (Kim [Pg 4 Par 3] The image conversion engine 325 replaces the transparent color of the transparent image with the background color or the background image and generates an opaque image and then performs an encoding operation with a high compression ratio opaque image storage method and outputs the result to the relevant temporary storage device 232. [Pg 2 Par 4] In the conventional transparent image storage method, there are an indirect method in which n colors of various basic colors representing an image are stored in a table called a palette, and a direct method in which color information is included in a pixel which is the smallest unit of the image.);
Ozcan teaches:
and determine mean squared error (MSE) loss for the respective images in RGB space such that SD model weights only visible pixels as important for alpha channel prediction (Ozcan [0069] where D refers to the discriminator network output, z.sub.label denotes the brightfield image of the chemically stained tissue, z.sub.output denotes the output of the generator network. The generator loss function balances the pixel-wise mean squared error (MSE) of the generator network output image with respect to its label, the total variation (TV) operator of the output image, and the discriminator network prediction of the output image, using the regularization parameters (λ, α) that are empirically set to different values).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang, Xu, and He with Kim and Ozcan. Replacing the background in images and determining a MSE, as in Kim and Ozcan, would benefit the Wang, Xu, and He teachings by allowing for a way to change the background in an image as well as finding a MSE. Additionally, this is the application of a known technique, changing the background of an image and finding a MSE, to yield predictable results.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Lin et al. (US 20210027470), Park et al. (KR 101665137), and Wang et al. (CN 116824004).
Regarding Claim 14:
Burgert and Lin teach:
A method comprising: extracting an alpha channel in images generated by a stable diffusion (SD) model to force the SD model to generate output images with respective backgrounds in a specified color (Burgert [Pg 3 Par 1] We illustrate how an input image and random background are alpha blended to generate a composite image. This image and its relevant text prompt are processed by our diffusion model based inference-time objective. Iterative gradient based optimization of the randomly initialized alpha mask converges to a segmentation optimal for the conditioning text prompt. Note that our diagram shows the alpha mask at an intermediate iteration: at the initial iteration it is entirely random Gaussian noise.)(Lin [0078] As shown in FIG. 6A, the image composition system 106 implements the training by providing an easy training foreground image 604 to the multi-level fusion neural network 616. The easy training foreground image 604 portrays a foreground object 610 against a pure color background 612. In one or more embodiments, the image composition system 106 generates the easy training foreground image 604 by compositing a matting image (i.e., containing the foreground object 610) from a matting dataset with the pure color background 612 using an alpha channel of the matting image.);
Lin fails to teach:
and removing the backgrounds in output post-processing at least in part by implementing a noise mask that covers and/or removes edges of the respective images,
applying noise and generating content only in the center of the respective images.
Park teaches:
and removing the backgrounds in output post-processing at least in part by implementing a noise mask that covers and/or removes edges of the respective images (Park [Pg 4 Par 4] If it is determined that the edge area is the center pixel, the noise removal filter 330 removes noise from the mask area, as in the case of the center pixel R or B, and the noise removal operation will be described later.),
Burgert, Guo, and Wang teach:
applying noise and generating content only in the center of the respective images (Wang [Pg 6 Par 4] The foreground mask image is a single-channel image, which can represent the position information of the interested foreground object in the image, wherein the area where the foreground object is located can be black, and the background area can be white.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Burgert with Lin, Park and Wang. Removing the background and applying a mask to remove the noise in the mask area, as in Lin, Park and Wang, would benefit the Burgert teachings by allowing for a way apply a noise mask to certain parts of an image. Additionally, this is the application of a known technique, removing backgrounds and applying a mask, to yield predictable results.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), Friedman et al. (US 20210019603).
Regarding claim 17:
Burgert, Guo, and Wang teach:
The apparatus of Claim 15,
Wang fails to teach:
wherein the instructions are executable to train the decoder: modifying a variational autoencoder (VAE) portion of the decoder to output a fourth channel encoding alpha information.
Friedman teaches:
wherein the instructions are executable to train the decoder: modifying a variational autoencoder (VAE) portion of the decoder to output a fourth channel encoding alpha information (Friedman [0037] In some examples, the encoder neural network 305 and the decoder neural network 310 are trained together using a variational autoencoder [0037] In some examples, the decoder neural network 310 generates α, β parameters as the probability model parameters 315 for a Binomial distribution to generate ratings.) (Friedman [0065] Example 5 includes the apparatus of example 1, wherein the training controller is to train the neural network at least in part with a variational autoencoder, the variational autoencoder including an encoder and a decoder.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang with Friedman. Training a decoder and modifying a VAE, as in Friedman, would benefit the Wang teachings by allowing for a way to create a custom VAE. Additionally, this is the application of a known technique, modifying a VAE with allowing an alpha channel, to yield predictable results.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), and He et al. (WO 2022271146).
Regarding claim 18:
Burgert, Guo, and Wang teach:
The apparatus of Claim 15,
Wang fails to teach:
wherein the instructions are executable to train the decoder using a dataset comprising plural RGBA images with transparent backgrounds and versions of the plural images generated by applying one or more of random flips, rotations, zooms, and color augmentations of the plural images.
He teaches:
wherein the instructions are executable to train the decoder using a dataset comprising plural RGBA images with transparent backgrounds and versions of the plural images generated by applying one or more of random flips, rotations, zooms, and color augmentations of the plural images (He [0060] In order to enhance the generalization potential of the decoder machine learning model 134a can augment the second dataset by generating new training samples using the existing training samples of the second dataset. To generate the new training samples, the training process can distort training images among the set of training images to create distorted images that are used to train the model. In some implementations, the distorted images can be generated by applying visual perturbations that widely occur in real-world visual data such as horizontal and vertical flips, translations, rotation, cropping, color distortions, adding random noise etc. In some implementations, the training process can generate new training samples by encoding the training images into different file formats using lossy compression or transformation techniques. [0043] For example, a 32-bit RGB pixel includes 8 bits for each color channel (e.g., Red (R), Green (G) and Blue (B)) and an “alpha” channel for transparency.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang with He. Having images that apply an effect to them, as in He, would benefit the Wang teachings by allowing for a wider set of images that may be used. Additionally, this is the application of a known technique, applying effects to images, to yield predictable results.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), He et al. (WO 2022271146), Kim et al. (KR 20160059488), and Dong et al (CN 113744280).
Regarding claim 19
Burgert, Guo, Wang and He teach:
The apparatus of Claim 18,
Wang and He fail to teach:
wherein the instructions are executable to: transform at least some of the RGBA images in the dataset into respective RGB images by replacing background in the some of the RGBA images with randomly generated, low-variance backgrounds;
Kim teaches:
wherein the instructions are executable to: transform at least some of the RGBA images in the dataset into respective RGB images by replacing background in the some of the RGBA images with randomly generated, low-variance backgrounds (Kim [Pg 4 Par 3] The image conversion engine 325 replaces the transparent color of the transparent image with the background color or the background image and generates an opaque image and then performs an encoding operation with a high compression ratio opaque image storage method and outputs the result to the relevant temporary storage device 232.);
input the respective RGB images into the decoder to cause the decoder to predict corresponding RGBA image.
Dong teaches:
input the respective RGB images into the decoder to cause the decoder to predict corresponding RGBA image (Dong [Pg 15 Par 8] the color image will have RGBA four channels, RGB respectively red, green; blue three basic color channels, A represents the Alpha channel, describing the transparency of the image, but in the real process, in the image shooting process, it will not record the Alpha information, therefore, the instance segmentation module predicts the Alpha information of the image.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang and He with Kim. Converting an RGBA to RGB, as in Kim, would benefit the Wang and He teachings by allowing to change images around. Additionally, this is the application of a known technique, converting RGBA to RGB, to yield predictable results.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Burgert et al. in view of Wang et al. (CN 116824004), Guo et al. (CN 116630464), He et al. (WO 2022271146), Kim et al. (KR 20160059488), and Ozcan et al (US 20210043331).
Regarding claim 20
Burgert, Guo, Wang He teach:
The apparatus of Claim 18,
Wang and He fail to teach:
wherein the instructions are executable to: replace transparent pixels in the at least some of the RGBA images with a random-colored background image to convert the at least some of the RGBA images back to respective images in RGB space;
and determine mean squared error (MSE) loss for the respective images in RGB space such that SD model weights only visible pixels as important for alpha channel prediction.
Kim teaches:
wherein the instructions are executable to: replace transparent pixels in the at least some of the RGBA images with a random-colored background image to convert the at least some of the RGBA images back to respective images in RGB space (Kim [Pg 4 Par 3] The image conversion engine 325 replaces the transparent color of the transparent image with the background color or the background image and generates an opaque image and then performs an encoding operation with a high compression ratio opaque image storage method and outputs the result to the relevant temporary storage device 232. [Pg 2 Par 4] In the conventional transparent image storage method, there are an indirect method in which n colors of various basic colors representing an image are stored in a table called a palette, and a direct method in which color information is included in a pixel which is the smallest unit of the image.);
Ozcan teaches:
and determine mean squared error (MSE) loss for the respective images in RGB space such that SD model weights only visible pixels as important for alpha channel prediction (Ozcan [0069] where D refers to the discriminator network output, z.sub.label denotes the brightfield image of the chemically stained tissue, z.sub.output denotes the output of the generator network. The generator loss function balances the pixel-wise mean squared error (MSE) of the generator network output image with respect to its label, the total variation (TV) operator of the output image, and the discriminator network prediction of the output image, using the regularization parameters (λ, α) that are empirically set to different values).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Wang and He with Kim and Ozcan. Converting an RGBA to RGB and finding MSE, as in Kim and Ozcan, would benefit the Wang and He teachings by allowing to change images around. Additionally, this is the application of a known technique, converting RGBA to RGB and finding MSE, to yield predictable results.
Response to Arguments
Applicant's arguments filed 2/17/2026 have been fully considered but they are not persuasive.
Applicant alleges:
“In regard to the 103 rejections, the Office relies on Burgert's Peekaboo model (page 1, paragraph 3, see page 3 of the Action) to teach "modify a stable diffusion (SD) model" as stated in claim 1. However, Burgert's section "5 Generating Images with Transparency" pseudocode indicates that the Peekaboo model is not being modified but, rather, the un-modified Peekaboo model is given new input data, namely the alpha mask (screenshot of pseudocode included below):
Despite these existing distinctions, and to help achieve compact prosecution, independent claims 1 and 15 have been amended and are allowable at least in view of these amendments. In addition, new claims 21 through 26 are allowable as the applied references do not disclose, teach or suggest at least their corresponding features.
The dependent claims are allowable for at least the same reasons as their respective independent claims. Because each claim is deemed to define additional aspects of the disclosure, the individual consideration of each claim on its own merits is respectfully requested. Withdrawal of all rejections is therefore respectfully requested.“
In response:
Burgert in the introduction mentions “Our proposed Peekaboo, based on a pre-trained image-language stable diffusion model [17], achieves unsupervised semantic and referring segmentation without having to training any model.”
This implies that the peekaboo model is based on the stable diffusion model meaning that some modification was done to stable diffusion.
Burgert teaches:
based on modifying a noise distribution component of the SD model (Burgert [Pg 4 Par 2] First introduced in DreamFusion [45], Score Distillation Sampling (SDS) is a method that generates samples from a diffusion model by optimizing a loss function we call score distillation loss (SDL). This allows us to optimize samples in any parameter space, as long as we can map back to images in a differentiable manner. We modify SDS to optimize learnable alpha masks and operate with latent diffusion. [Pg 4 Par 7] The Stable Diffusion model jointly processes images and text. Its visual encoder first projects images
to a latent space. This latent vector is then processed by the diffusion U-Net (D) conditioned on text embedding (from text encoder T ) to produce noise outputs. To measure Ls, we first degrade latent vector z of composite image x using forward diffusion, introducing Gaussian noise ϵ ∼ N to z, resulting in a noisy ˜z. We then perform diffusion denoising conditioned on the text embedding with pre-trained D. Our loss Ls is measured as the reconstruction error of noise ϵ, given noisy ˜z and text embedding T (p) as in Eq. (2): Ls = MSE(ϵ,D(˜z,T (p))) (2)
where MSE refers to mean-squared loss. Pseudo-code describing Ls in detail is presented in Algorithm 1)
Gou teaches:
and using a decoder of the SD model that is trained separately from a corresponding encoder of the SD model that forms a variational autoencoder (VAE) with the decoder of the SD model (Gou [ABSTRACT] An image generation network is constructed (S102) based on a stable diffusion model. An image encoder and an image decoder are trained (S103) by a low-rank adaptation method in combination with original image samples and image description text in the pre-training data set to obtain a positive sample generation model. A controlled image generation network is constructed (S104). The image encoder and the image decoder are trained (S105) by using a low-rank adaptation method to obtain a negative sample generation model)
Burgert mentions how the noise of the model specifically is modified in a way. Specifically, by using denoising it modifies the noise.
Gou teachers that the LORA method is used and both the encoder and decoder are trained. The LORA method while most commonly used for training the encoder and decoder simultaneously can be used separately to allow for both of the encoder and decoder to be trained separately.
As well, claims 15, 18, and 21-16 are rejected:
modify an encoded image to include (i) first noise values within a first region of the encoded image and (ii) second noise values within a second region of the encoded image, wherein the first noise values and the second noise values represent different statistical distributions of randomized noise; generating, using (i) the modified encoded image and (ii) the SD model trained using output of a second model, different than the SD model, and a plurality of training images representing plain colored backgrounds each paired with a text prompt indicating the corresponding plain background, a latent representation of the modified encoded image; and generating, using the generated latent representation and the decoder,
for introducing new matter.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENIS VASILIY MINKO whose telephone number is (571)270-5226. The examiner can normally be reached Monday-Thursday 8:30-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENIS VASILIY MINKO/Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights