DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement\
The information disclosure statement (IDS) submitted on 01/02/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Pathak, Deepak, et al. ("Context encoders: Feature learning by inpainting." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016) in view of Avrahami et al. ("Blended diffusion for text-driven editing of natural images." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, as provided).
Regarding Claim 1,
Pathak discloses A method comprising: accessing a first image; (Pathak, Abstract, discloses an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders– a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s); image is obtained)
receiving a selected area of the first image and a prompt, wherein the prompt is indicative of a visual element; (Pathak, Introduction, pg. 2537, Fig. 1, discloses On the encoder side, we show that encoding just the context of an image patch and using the resulting feature to retrieve nearest neighbor contexts from a dataset produces patches which are semantically similar to the original (unseen) patch. We further validate the quality of the learned feature representation by fine-tuning the encoder for a variety of image understanding tasks, including classification, object detection, and semantic segmentation. We are competitive with the state-of-the-art unsupervised/self supervised methods on those tasks. On the decoder side, we show that our method is often able to fill in realistic image content. Indeed, to the best of our knowledge, ours is the first parametric inpainting algorithm that is able to give rea sonable results for semantic hole-filling (i.e. large missing regions). The context encoder can also be useful as a better visual feature for computing nearest neighbors in non parametric inpainting methods; image patch context (prompt to visual element) of an image (image patch))
determining an encoding of the prompt; (Pathak, Introduction, pg. 2537, Fig. 1, discloses On the encoder side, we show that encoding just the con text of an image patch and using the resulting feature to retrieve nearest neighbor contexts from a dataset produces patches which are semantically similar to the original (un seen) patch. We further validate the quality of the learned feature representation by fine-tuning the encoder for a variety of image understanding tasks, including classification, object detection, and semantic segmentation. We are competitive with the state-of-the-art unsupervised/self supervised methods on those tasks. On the decoder side, we show that our method is often able to fill in realistic image content. Indeed, to the best of our knowledge, ours is the first parametric inpainting algorithm that is able to give rea sonable results for semantic hole-filling (i.e. large missing regions). The context encoder can also be useful as a better visual feature for computing nearest neighbors in non parametric inpainting methods; image patch context (prompt to visual element) of an image (image patch) and visual image patch (element) context (prompt) is encoded)
generating a first visual noise based on the selected area of the first image; (Pathak, Fig. 1, (Pathak, Introduction, pg. 2537, Fig. 1, discloses On the encoder side, we show that encoding just the con text of an image patch and using the resulting feature to retrieve nearest neighbor contexts from a dataset produces patches which are semantically similar to the original (un seen) patch. We further validate the quality of the learned feature representation by fine-tuning the encoder for a variety of image understanding tasks, including classification, object detection, and semantic segmentation. We are competitive with the state-of-the-art unsupervised/self supervised methods on those tasks. On the decoder side, we show that our method is often able to fill in realistic image content. Indeed, to the best of our knowledge, ours is the first parametric inpainting algorithm that is able to give rea sonable results for semantic hole-filling (i.e. large missing regions). The context encoder can also be useful as a better visual feature for computing nearest neighbors in non-parametric inpainting methods; image patch context (prompt to visual element) of an image (image patch))
determining an encoding of the prompt; (Pathak, Introduction, pg. 2537, Fig. 1, discloses On the encoder side, we show that encoding just the con text of an image patch and using the resulting feature to retrieve nearest neighbor contexts from a dataset produces patches which are semantically similar to the original (un seen) patch. We further validate the quality of the learned feature representation by fine-tuning the encoder for a variety of image understanding tasks, including classification, object detection, and semantic segmentation. We are competitive with the state-of-the-art unsupervised/self supervised methods on those tasks. On the decoder side, we show that our method is often able to fill in realistic image content. Indeed, to the best of our knowledge, ours is the first parametric inpainting algorithm that is able to give rea sonable results for semantic hole-filling (i.e. large missing regions). The context encoder can also be useful as a better visual feature for computing nearest neighbors in non-parametric inpainting methods; image patch context (prompt to visual element) of an image (image patch) and visual image patch (element) context (prompt) is encoded and inpainted by filling the encoded image patch regions)
performing a first inpainting process on the selected area of the first image, based on the first visual noise and the encoding, to generate a second image, wherein the second image comprises a first representation of the visual element; (Pathak, Fig. 1-6, Random region completely remove those boundaries, we experimented with removing arbitrary shapes from images, obtained from random masks in the PASCAL VOC2012 dataset. We deform those shapes and paste in arbitrary places in the other images (not from PASCAL), again covering up to 1 4 of the image. Note that we completely randomize the region masking process, and do not expect or want any correlation between the source segmentation mask and the image. We merely use those regions to prevent the network from learning low-level features corresponding to the removed mask. See example in Figure 3c. In practice, we found region and random block masks produce a similarly general feature, while significantly outperforming the central region features. We use the random region dropout for all our feature-based experiments; 5.1. Semantic Inpainting We train context encoders with the joint loss function de fined in Equation (3) for the task of inpainting the missing region. However, the bottleneck is of 4000 units (in contrast to 100); see supplementary material. We used the default solver hyper-parameters suggested. We use λrec = 0.999 and λadv = 0.001. However, a few things were crucial for training the model. We did not condition the adversarial loss (see Section 3.2) nor did we add noise to the encoder. We use a higher learning rate for con text encoder (10 times) to that of adversarial discriminator. To further emphasize the consistency of prediction with the context, we predict a slightly larger patch that overlaps with the context (by 7px). During training, we use higher weight (10×) for the reconstruction loss in this overlapping region. The qualitative results are shown in Figure 4. Our model performs generally well in inpainting semantic regions of an image. However, if a region can be filled with low level textures, texture synthesis methods, such as [2, 11], can often perform better (e.g. Figure 5). For semantic in painting, we compare against nearest neighbor inpainting (which forms the basis of Hays et al. [19]) and show that our reconstructions are well-aligned semantically, as seen on Figure 6. It also shows that joint loss significantly im proves the inpainting over both reconstruction and adversarial loss alone. Moreover, using our learned features in a nearest-neighbor style inpainting can sometimes improve results over a hand-designed distance metrics. Table 1 reports quantitative results on StreetView Dataset 5.2. Feature Learning For consistency with prior work, we use the AlexNet architecture for our encoder. The networks were trained with a constant learning rate of 10−3 for the center-region masks. However, for random region corruption, we found a learning rate of 10−4 to perform better. We apply dropout with a rate of 0.5 just for the channel-wise fully connected layer, since it has more parameters than other layers and might be prone to overfitting. The training process is fast and converges in about 100K iterations: 14 hours on a Titan X GPU. Figure 7 shows inpainting results for context encoder trained with random region corruption using reconstruction loss. To evaluate the quality of features, we find nearest neighbors Figure 7: Arbitrary region inpainting for context encoder trained with reconstruction loss. to the masked part of image just by using the features from the context, see Figure 8. Note that none of the methods ever see the center part of any image, whether a query or dataset image. Our features retrieve decent nearest neighbors just from context, even though actual prediction is blurry with L2 loss. AlexNet features also perform decently as they were trained with 1M labels for semantic tasks, HOG on the other hand fail to get the semantics; selected image region (image patch) encoded with context (prompt related to image patch) is replaced with inpainting and second image is created)
Pathak does not explicitly disclose generating, based on the first image and the second image, a third image, the third image comprising a second representation of the visual element; generating, based on the selected area and a noise strength parameter, a second visual noise; performing a second inpainting process on an area of the third image corresponding to the selected area, based on the second visual noise and the encoding, to generate a fourth image, the fourth image comprising a third representation of the visual element
Avrahami discloses generating, based on the first image and the second image, a third image, the third image comprising a second representation of the visual element; generating, based on the selected area and a noise strength parameter, a second visual noise; performing a second inpainting process on an area of the third image corresponding to the selected area, based on the second visual noise and the encoding, to generate a fourth image, the fourth image comprising a third representation of the visual element. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, inpu tmask m, and a text promptd, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ theelement-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Pathak in view of Avrahami having a method of encoding context of selected image region (patch) in an image for inpainting, with the teachings of Avrahami having, by the training module, performing blending of pixels according to non-selected regions from first, second, third and fourth image into selected image regions by process of inpainting of text prompted image regions by providing noise strength values and replacing selected image regions by inpainting accordingly in order to fill the missing parts of the image for visual clarity.
Regarding Claim 2,
The combination of Pathak and Avrahami further discloses generating a final image, by inserting at least a portion of the fourth image into an area of the first image that corresponds with the user selected area; wherein the portion of the fourth image comprises the third representation of the visual element. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, inpu tmask m, and a text promptd, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ theelement-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 3,
The combination of Pathak and Avrahami further discloses wherein the portion of the fourth image that is inserted into the first image is an area of the fourth image that corresponds with the user selected area. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 4,
The combination of Pathak and Avrahami further discloses providing an output, wherein the output is the fourth image and/or the final image. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 5,
The combination of Pathak and Avrahami further discloses wherein the output is provided by one or more of: displaying the output on a display; sending the output to another device; producing a print out of the output; and/or saving the output to a computer-readable storage medium. (Avrahami, pg. 15-17, B.3. Inference time, discloses synthesis time for a single image using one NVIDIA A10 GPU: Our method (Algorithm 2) & Local CLIP-guided diffusion (Algorithm D: 27 seconds. PaintByWord+ +: 78 seconds. Original paint by word did not release their code and did not mention the run-time. we generate several results for the same inputs and use the best ones. Instead of generating them sequentially, we accelerate the generation process using two techniques; 2. Parallel generation: Because each of the generation processes is independent, we can distribute the generation across multiple GPUs. concurrently used 4 NVIDIA A10 GPUs; GPUs process, save and output results). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 6,
The combination of Pathak and Avrahami further discloses wherein generating the third image comprises blending the first image with the second image based on a blending factor, wherein the blending of the first image with the second image is based on the equation: Third image = second image x blending factor + first image x (1 - blending factor), wherein the blending factor is a value between 0.0 to 1.0.
Regarding Claim 7,
The combination of Pathak and Avrahami further discloses wherein part of the first and second inpainting processes are performed by:using a machine learning or artificial intelligence model that is a diffusion model, and/or an Artificial Neural Network, and/or a fully convolutional neural network, and/or. a U-Net, and/or using Stable Diffusion. (Pathak, pg. 2536, Introduction, discloses
PNG
media_image1.png
200
400
media_image1.png
Greyscale
; artificial neural network is used to inpainting process). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 8,
The combination of Pathak and Avrahami further discloses wherein the second representation of the visual element is a semi-transparent version of the first representation of the visual element. (Pathak, pg. 2536, Fig. 1(a-d), discloses input image and qualitative illustration of the task given an image with an image region (b) automatic inpainting by context encoder; semitransparent image of input image region (visual element of windows) is inpainted). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 9,
The combination of Pathak and Avrahami further discloses wherein an area of the first image that corresponds with the selected area comprises pixel information, and wherein generating the first visual noise comprises adding signal noise to the pixel information, and/or wherein an area of the third image that corresponds with the selected area comprises pixel information, wherein the pixel information is indicative of the second representation of the visual element; and wherein generating the second visual noise comprises adding signal noise based on the noise strength parameter to the pixel information. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4 where lambda (signal noise) is strength parameter). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 10,
The combination of Pathak and Avrahami further discloses wherein adding signal noise based on the noise strength parameter to the pixel information of the area of the third image that corresponds with the selected area comprises increasing or decreasing the amount of signal noise based on the value of the noise strength parameter. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4 where lambda (signal noise) is strength parameter). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 11,
The combination of Pathak and Avrahami further discloses wherein the pixel information is a mapping of pixel information to a lower-dimensional latent space. (Avrahami, pg. 2, Text-driven image manipulation, discloses utilize CLIP in order to manipulate real images. Style CLIP use pretrained StyleGAN2 and CLIP models to modify images based on text prompts. To manipulate real images (rather than generated ones), they must first be encoded to the latent space. This approach can not handle generic real images, and is restricted to domains for which high-quality generators are available. In addition, StyleCLIP operates on images in a global fashion, without providing spatial control over which areas should change; mapped to latent space). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 12,
The combination of Pathak and Avrahami further discloses wherein each of the images and each of the representations of the visual element comprise one or more visual attributes; and wherein at least one of the visual attributes of the third representation is more similar to the first image than the corresponding visual attribute of the first representation. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4 where lambda is strength parameter). (Avarahami, Fig. 1). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 13,
The combination of Pathak and Avrahami further discloses wherein the at least one attribute comprises one or more of:colour; colour model; texture; brightness; shading; dimension; bit depth; hue; saturation; and/or lightness. (Avarahami, Fig. 1 discloses one of pixel color, shading, texture, hue or brightness). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 14,
The combination of Pathak and Avrahami further discloses wherein the prompt is one of a:text string; audio recording; or image file, wherein when the prompt is a text string, determining an encoding of the prompt comprises: providing the text string to a text encoder. (Avrahami, pg. 2, Text-driven image manipulation, discloses utilize CLIP in order to manipulate real images. Style CLIP use pretrained StyleGAN2 and CLIP models to modify images based on text prompts. To manipulate real images (rather than generated ones), they must first be encoded to the latent space. This approach can not handle generic real images, and is restricted to domains for which high-quality generators are available. In addition, StyleCLIP operates on images in a global fashion, without providing spatial control over which areas should change; text prompt is encoded). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 15,
The combination of Pathak and Avrahami further discloses wherein the text encoder is a contrastive language- image pre-training (CLIP) text encoder. (Avrahami, pg. 2, Text-driven image manipulation, discloses utilize CLIP in order to manipulate real images. Style CLIP use pretrained StyleGAN2 and CLIP models to modify images based on text prompts. To manipulate real images (rather than generated ones), they must first be encoded to the latent space. This approach can not handle generic real images, and is restricted to domains for which high-quality generators are available. In addition, StyleCLIP operates on images in a global fashion, without providing spatial control over which areas should change; text prompt is encoded using CLIP). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 16,
The combination of Pathak and Avrahami further discloses the step of: determining, based on the user selected area, a cropped area, wherein the user selected area is entirely comprised within the cropped area; treating the cropped area as the first image for the steps of generating the first visual noise, performing the first inpainting process and generating the third image; and inserting the fourth image into the first image at the location corresponding to the cropped area to generate an output image. (Avrahami, Fig. 1-2, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4 where lambda is strength parameter; (Pathak, Fig. 1 discloses pixel values from non selected regions and selected regions are approximated with use of context encoders and pixel values in surrounding regions that is non selected regions by melding or blending of pixels process). Avarahami, Fig. 1 Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 17,
The combination of Pathak and Avrahami further discloses wherein the cropped area comprises a non-selected region, wherein the non-selected region is a region of the cropped area that is not within the user selected area. (Avrahami,Fig. 1-2, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to “fool” CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4 where lambda is strength parameter; (Pathak, Fig. 1 discloses pixel values from non selected regions and selected regions are approximated with use of context encoders and pixel values in surrounding regions that is non selected regions by melding or blending of pixels process). (Avarahami, Fig. 1). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 18,
The combination of Pathak and Avrahami further discloses subsequent to generating the fourth image, performing a melding process, wherein the melding process comprises; blending pixel information of the fourth image with pixel information of the first image; wherein the melding process results in the pixel information of the fourth image subsequent to the melding process being more similar to the pixel information of the first image, than the pixel information of the non-selected regions prior to the melding process. (Avrahami, Fig. 3. Effect of λ in localCLIP-guided diffusion. Given an input image with a mask and the prompt “a dog”: with λ set too low(λ=100), the entire image changes completely, while if λ is too high(λ=10000), the model fails to change the foreground (and the background preservation is not perfect). Using an intermediate value(λ=1000)the model changes the foreground while resembling the original background(zoom for more details)., Fig. 4. Text-driven blended diffusion. Given input image x, input mask m, and a text prompted, we leverage the diffusion process to edit the image locally and coherently. We denote with⊙ the element-wise blending of two images using the input mask m; 4. Method , 4.1. Local CLIP-guided diffusion, 4.2.Text-drivenblendeddiffusion Given an image x, a guiding text prompt d and a binary mask m that marks the region of interest in the image, our goal is to produce a modified image x, s.t. the content x⊙m is consistent with the text description d, while the complementary area remains as close as possible to the source image, i.e., x⊙(1−m) ≈ x⊙(1−m),where ⊙is element-wise multiplication. Furthermore, the transition between the two areas of x should ideally appear seamless. In Section 4.1 we start by adapting the DDPM approach described above to incorporate local text-driven editing by adding a guiding loss comprised of a masked CLIP loss and a background preservation term. The resulting method still falls short of satisfying our requirements, and we proceed to present a new text-driven blended diffusion method in Section 4.2, which guarantees background preservation and improves the coherence of the edited result. Section 4.2.2 introduces an augmentation technique that we employ in or der to avoid adversarial results. 4.1. Local CLIP-guided diffusion; a pretrained CLIP model may be used to guide diffusion towards a target prompt. Since CLIP is trained on clean images (and retraining it on noisy images is im practical), we need a way of estimating a clean image x0 from each noisy latent xt during the denoising diffusion process. A similar approach is used in CLIP-guided diffusion [8], where a linear combination of xt and x0 is used to provide global guidance for the diffusion. The guidance can be made local, by considering only the gradients of DCLIP under the input mask. In this manner, we effectively adapt CLIP-guided diffusion [8] to the local (region-based) editing setting; Inspired by this technique, we propose to perform the blending at different noise levels along the diffusion process. Our key hypothesis is that at each step during the diffusion process, a noisy la tent is projected onto a manifold of natural images noised to a certain level. While blending two noisy images (from the same level) yields a result that likely lies outside the mani fold, the next diffusion step projects the result onto the next level manifold, thus ameliorating the incoherence. Thus, at each stage, starting from a latent xt, we perform a single CLIP-guided diffusion step, that denoises the latent in a direction dependent on the text prompt, yielding a latent denoted xt−1,fg. In addition, we obtain a noised version of the background xt−1,bg from the input image using Equation (2). The two latents are now blended using the mask: xt−1 = xt−1,fg ⊙ m+xt−1,bg ⊙(1−m), and the process is repeated (see Figure 4 and Algorithm 2). In the final step, the entire region outside the mask is re placed with the corresponding region from the input image, thus strictly preserving the background. 4.2.2 Extending augmentations Adversarial examples is a well-known phenomenon that may occur when optimizing an image directly on its pixel values. For example, a classifier can be easily fooled to classify an image incorrectly by slightly altering its pixels in the direction of their gradients with respect to some wrong class. Adding such adversarial noise will not be perceived by a human, but the classification will be wrong. Similarly, gradual changes of pixel values by CLIP guided diffusion, might result in reducing the CLIP loss without creating the desired high-level semantic change in the image. We find that this phenomenon frequently occurs in practice. experienced this issue and addressed it using a non-gradient method that is based on evolution strategy. We hypothesized that this problem can be mitigated by performing several augmentations on the intermediate result estimated at each diffusion step, and calculating the gradients using CLIP on each of the augmentations separately. This way, in order to fool CLIP, the manipulation must do so on all the augmentations, which is harder to achieve with out a high-level change in the image. Indeed, we find that a simple augmentation technique mitigates this problem: given the current estimated result x0, instead of taking the gradients of the CLIP loss directly, we compute them with respect to several protectively transformed copies of this image. These gradients are then averaged together. We term this strategy as “extending augmentation”. The effect of these augmentations is studied in Section 5.2. We’ve added extending augmentations to our method (Algorithm 2) as well as to the Local CLIP GD baseline (Algorithm 1) for all the comparisons in this paper. 4.2.3 Result ranking Algorithm 2 can generate multiple outputs for the same input; this is a desirable feature because our task is one-to many by its nature. we found it beneficial to generate multiple predictions, rank them and choose those with the higher scores. For the ranking, we utilize the CLIP model using the same DCLIP from Equation (6) on the final results, without the extending augmentations; Fig. 3 and Fig. 4 discloses inpainting of using various noise strength (lambda values) and changing the visual select element being first, second, third, fourth and onwards according to visual clarity and relacing the prompted image areas according to created blended or diffused first, second, third, fourth and so on images as shown in fig. 3 and fig. 4 where lambda is strength parameter). (Pathak, Fig. 1 discloses pixel values from non selected regions and selected regions are approximated with use of context encoders and pixel values in surrounding regions that is non selected regions by melding or blending of pixels process). (Avarahami, Fig. 1). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Regarding Claim 20,
The combination of Pathak and Avrahami further discloses A non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform the method of claim 1. (Avrahami, pg. 15-17, B.3. Inference time, discloses synthesis time for a single image using one NVIDIA A10 GPU: Our method (Algorithm 2) & Local CLIP-guided diffusion (Algorithm D: 27 seconds. PaintByWord+ +: 78 seconds. Original paint by word did not release their code and did not mention the run-time. we generate several results for the same inputs and use the best ones. Instead of generating them sequentially, we accelerate the generation process using two techniques; 2. Parallel generation: Because each of the generation processes is independent, we can distribute the generation across multiple GPUs. concurrently used 4 NVIDIA A10 GPUs; GPUs process, save and output results). Additionally, the rational and motivation to combine the references Pathak and Avrahami as applied in rejection of claim 1 apply to this claim.
Allowable Subject Matter
Claim 19 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US-20220129682-A1 (Tang et al., Methods and systems for fully-automatic image processing to
detect and remove unwanted people from a digital image of a photograph. The system includes the following modules: 1) Deep neural network (DNN)-based module for object segmentation and head pose estimation; 2) classification (or grouping) of wanted versus unwanted people based on information collected in the first module; 3) image inpainting of the unwanted people in the digital image. The classification module can be rules-based in an example. In an example, the DNN-based module generates, from the digital image: 1. A list of object category labels, 2. A list of object scores, 3. A list of binary masks, 4. A list of object bounding boxes, 5. A list of crowd instances, 6. A list of human head bounding boxes, and 7. A list of head poses (e.g., yaws, pitches, and rolls), Abstract)
US-20220083806-A1 (Cho et al., A method and a system for performing product search based on image restoration receive an input image including a target product to be searched; detect one or more objects including the target product from the input image through object detection; determine the target product among a plurality of products and obtain a main product image representing the target product; determine whether an obstacle, which is an object other than the target product, is included in the main product image; when the obstacle is included in the main product image, generate a loss image by removing the obstacle from the main product image; obtain a restored image by performing image restoration on the loss image using the deep learning; perform a product search, which searches for a product similar to the target product, based on the restored image; and output a result of the product search, Abstract)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PINALBEN V PATEL whose telephone number is (571)270-5872. The examiner can normally be reached M-F: 10am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Chineyere Wills-Burns can be reached at 571-272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Pinalben Patel/Examiner, Art Unit 2673
Prosecution Insights