CTNF 18/956,284 CTNF 101720 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 102 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15 AIA Claim s 1, 9, are rejected under 35 U.S.C. 102( a)(1 ) as being anticipated by Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) . Regarding claim 1, 9, Li teaches: A method comprising: obtaining an input image and an input mask, wherein the input mask indicates a region of the input image to be modified; (Li 3 “Let us start to define an input RGB masked image and a binary mask image as I in ∈ R H×W× 3 and M ∈ R H×W respectively. The pixel values input to our model are normalized between 0 and 1 and pixels with value 1 in M represent the masked regions. I coarse ∈ R H×W× 3 denotes the output of our coarse generator G 1 at the first coarse reconstruction stage.” 1 “In this paper, we present a coarse-to-fine Deep Generative Inpainting Network (DeepGIN) which consists of two stages, namely coarse reconstruction stage and refinement stage. Similar to the network design of previous studies [20,33,34], the coarse reconstruction stage is responsible for rough estimation of the missing pixels in an image while the refinement stage is responsible for detailed decoration on the coarse reconstructed image.” Note: Li teaches that a model accepts an RGB image with an associated mask for the image as inputs. From these inputs in the first stage the coarse generator G1 generates an output image where the area that has been masked has its missing pixels estimated, teaching that the mask indicates a region of the input image to be modified.) generating, using a first image generation model, an intermediate result based on the input image and the input mask, wherein the intermediate result modifies the region of the input image indicated by the input mask; (Li 1, cited previously, teaches that the first stage of the model will generate an image where the masked area has its missing area generated by the model. As this produced image is only the output of G1 the coarse generator, and will be fed to the refinement generator later Li teaches an intermediate result based on the input image and mask where the intermediate result modifies the masked region.) and generating, using a second image generation model, a synthetic image based on the input image and the intermediate result, wherein the synthetic image depicts the input image with content from the modified region at a higher level of detail than the intermediate result. (Li 4 “Our proposed Deep Generative Inpainting Network (DeepGIN) consists of two stages as shown in Fig. 2, a coarse reconstruction stage and a refinement stage . The first coarse generator G 1( I in, M ) is trained to roughly reconstruct the masked regions and gives I coarse . The second refinement generator G 2( I coarse, M ) is trained to exquisitely decorate the coarse prediction with details and textures , and eventually forms the completed image I out ( I compltd ). For our discriminators, motivated by SN-GANs [19,34] and multi-scale discriminators [10,28], we modify and employ two SN-GAN based discriminators D ( I in, I compltd ) which operate at two image scales, 256 × 256 and 128 × 128 respectively, to encourage better details and textures of local reconstructed patterns at different scales. Details of our network architecture and learning are shown below.” Note: Li teaches that after the intermediate result, the output of the coarse generator, is obtained, the refinement generate generates a final output image with a higher level of detail and texture.) Regarding claim 9, Li teaches: The method of claim 1, wherein generating the synthetic image comprises: inpainting the region indicated by the input mask with content consistent with the input image. (Li 1 “Image inpainting (also called image completion) is a task of predicting the values of missing pixels in a corrupted/masked image such that the completed image looks realistic and is semantically close to the reference ground truth even though it does not exist in real-world situations” PNG media_image1.png 852 1272 media_image1.png Greyscale Note: Li’s DeepGin seeks to inpaint the masked regions, which it defines as attempting to realistically recreate the missing space as close to the original image as possible. Thus, DeepGin teaches that inpainting regions indicated by a mask are inpainted with content consistent with the input.) Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 2 is rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of He (Mask R-CNN) . Regarding claim 2, Li teaches: The method of claim 1, wherein obtaining the input mask comprises: segmenting the input image ( PNG media_image2.png 922 1020 media_image2.png Greyscale Note: While Li has already been shown to include a mask of an image in as an input, teaching that the image has been segmented into a mask/non masked portion, Fig. 6 shows examples of an image segmentation with masks the model will fill.) Li teaches that through segmentation an image can be divided into a masked/non masked portions to provide with the input image to provide to first image generation model. Li does not however teach that elements/objects are identified as part of performing segmentation. This is taught by He segmenting the input image to identify an element of the input image. (He Abstract “We present a conceptually simple, flexible, and general framework for object instance segmentation . Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition.”) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with He where a first model that accepts an input image with a segmented mask identifies the element/object in the segmented area. There are several reasons that would motivate one to do so, rather than generating an entirely new area over a broader area that has been masked/segmented a user may wish to remove or change the appearance of just the element/object in the area. By identifying what an element is the generative model can more accurately remove the element, change its appearance, etc… resulting in a higher quality generation . 07-21-aia AIA Claim s 3, 6, 7 , 12, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Ramesh (US 11983806 B1) . Regarding claim 3, Li teaches: The method of claim 1, wherein obtaining the input mask comprises: While Li teaches accepting an input mask for the input image it does not detail that the mask is determined by a location input. This is found in Ramesh which teaches receiving a location input; and generating the input mask based on the location input. (Ramesh Col. 10 Line 1 “For example, removing the masked region may involve masking the pixel values in the masked region by replacing the pixel values with a mask of pixels with a value of zero. In some embodiments, the masked region may be determined on a graphical user interface. In some examples, one or more portions of the input image may be identified to be masked. For example, the masked region may be selected by a user via a user interface, such as using a mouse to outline the region to be removed.” Note: The specifications define a location input as ¶34 “and system further include receiving a location input. Some examples further include generating the input mask based on the location input” ¶69 “In some cases, a user may provide a rough sketch indicating the location of the object to be removed. Then, machine learning model 515 may generate a precise mask representing the object based on the rough sketch.” The specifications define a location input as a means to generate the input mask, providing an example of a providing a drawing that is used to determine the mask. This same idea is taught by Ramesh which teaches determining a mask based on a user’s mouse outline/drawing.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Ramesh where a mask input that is provided alongside an input image to a first model can determine the mask based on a location input. There are several reasons that would motivate one to do so, if a user is interacting with the described system, it may be tedious and difficult to precisely define the edges of an object or area they wish to remove, by allowing the system to determine the mask based on an easier to provide location input, such as a rough drawing, users can more easily and accurately mask specific areas. Regarding claim 6, Li teaches: The method of claim 1, wherein: Li does not however teach that an element masked by the input mask should be removed instead of reconstructed, this is taught by Ramesh which teaches the input mask indicates an element of the input image and the synthetic image removes the element from the input image. (Ramesh Col. 10 Line 24 “It will be appreciated that input images may include a variety of styles, including realistic images, photographs, digital art, oil paintings, and the like. Input image 200 may depict a variety of objects which may be edited, altered, or changed, such as plate object 202. For example, it may be desired to remove object 202, edit various features of object 202 or replace object 202. FIG. 3 illustrates an example of a masked image 300, consistent with embodiments of the present disclosure. Masked image 300 may be a version of input image 200 such that masked region 302 represents a masked portion of input image 200. For example, editing object 202 in input image 200 may involve determining the masked region 302. In some embodiments, masked region 302 may be determined by selecting or outlining the area with a tool, such as eraser 110 in dashboard” Note: Ramesh teaches objects in an image can be removed, edited, or replaced, and that these objects to be edited/removed are determined by being in the masked region.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Ramesh where a model that accepts an input image and corresponding mask can remove the element in the masked region. There are several reasons that would motivate one to do so, many users may wish to remove objects or elements from an image, a common feature in photo editing software, this feature could be easily implemented to the described invention if when generating the masked area, the model chooses to generate empty space of the surrounding areas rather than reconstructing the object present in the non-masked image. Regarding claim 7, Li teaches: The method of claim 1, wherein generating the intermediate result comprises: While Li teaches using generative models it does not specify that one if its models is a diffusion model capable of performing reverse diffusion, this is found in Ramesh which teaches obtaining a first noise input; (Ramesh Col. 12 Line 4, cited below, teaches that a noise input is added to the input as part of the reverse diffusion process.) and denoising the first noise input to obtain the intermediate result. (Ramesh Col. 12 Line 4 “In some embodiments, the machine learning model may include a diffusion model . A diffusion model may involve a transformer-implemented generative model … For example, a diffusion model may involve a neural network which denoises an image by reversing a diffusion process. The diffusion process may involve adding noise to an input , such as randomly sampled noise and Gaussian noise, at different steps, until the input becomes indistinguishable from noise. In some embodiments, the process of adding noise may include a Markov chain beginning with an image and resulting in an image that is an approximation of pure noise. The diffusion model may be trained to reverse the noising process and attempt to reconstruct the noisy image to its original form.” “The first sub-model may generate an image embedding, such as an image embedding corresponding to the masked image. Some disclosed embodiments involve a second sub-model configured to generate the enhanced image based on at least one of the image embedding, the text input, the masked image, and the masked region. The second sub-model may include a decoder and/or a diffusion model , as described herein. For example, the second sub-model may be provided the image embedding and may generate an output , such as the enhanced image, based on the image embedding. It will be appreciated that using image embeddings for the second sub-model to create the enhanced image by replacing pixel values in the masked region provides various advantages to the generated image, including producing more realistic images” Col. 15 Line 3 “In some examples, step 410 may include the generation of an image segment, such as pixel values corresponding to a masked region of an image, including image regions inside and outside an image … In some embodiments, step 410 of generating outputs may include iteratively regenerating an image, such as using generated enhanced images as inputs to the machine learning model.” Note: Ramesh teaches that from an input masked image a diffusion model will generate an output where the pixel values in the masked region are generated. This is the models “intermediate result” as Col. 15 Line 3 teaches that the output image may be iteratively regenerated to enhance the image, meaning it is an intermediate result not the final result.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Ramesh where a system that generates an intermediate result is able to do so via adding noise and denoising the input to produce the generated image. There are several reasons that would motivate one to do so, diffusion models are a popular type of generative model known to excel at accurate image generation tasks, a higher quality intermediate result could be obtained by leveraging a diffusion models high quality output. Regarding claim 12, Li teaches: A non-transitory computer readable medium storing code for image processing, the code comprising instructions that, when executed by at least one processor, (Li 5.1 “We developed our model using Pytorch 1.5.0 [22] and trained it on two NVIDIA GeForce RTX 2080Ti GPUs.” Note: Li teaches use of a 2080Ti, a processor that has its own onboard storage medium.) cause the at least one processor to perform operations comprising: obtaining an input image and an input mask, wherein the input mask indicates an element of the input image to be modified; (Li 3, cited in claim 1, teaches accepting an input image along with an input mask indicating an area to be generated over. Li does not however teach that the masked potion indicates an element or object to be removed.) generating, using a first image generation model, an intermediate result based on the input image and the input mask, (Li 1, cited previously, teaches that the first stage of the model will generate an image where the masked area has its missing area generated by the model. As this produced image is only the output of G1 the coarse generator, and will be fed to the refinement generator later, Li teaches an intermediate result based on the input image and mask where the intermediate result modifies the masked region.) wherein the intermediate result modifies the element of the input image indicated by the input mask; and generating, using a second image generation model, a synthetic image based on the input image and the intermediate result, wherein the synthetic image depicts the input image without the element removed in the intermediate result. (Li 4, cited in claim 1,teaches that after the intermediate result, the output of the coarse generator, is obtained, the refinement generate generates a final output image with a higher level of detail and texture.) While Li teaches a two-model system where a first model accepts an input image with an input mask and seeks to generate content to fill the masked region it does not specify that the element in the masked region should be removed. This is taught by Ramesh which teaches obtaining an input image and an input mask, wherein the input mask indicates an element of the input image to be removed; (Ramesh Col. 10 Line 24, cited below, teaches accepting an input image with a masked region where the element/object in the masked region can be removed, or alternatively edited/replaced.) wherein the result removes the element of the input image indicated by the input mask; wherein the synthetic image depicts the input image without the element removed ( Ramesh Col. 10 Line 24 “It will be appreciated that input images may include a variety of styles, including realistic images, photographs, digital art, oil paintings, and the like. Input image 200 may depict a variety of objects which may be edited, altered, or changed, such as plate object 202. For example, it may be desired to remove object 202, edit various features of object 202 or replace object 202. FIG. 3 illustrates an example of a masked image 300, consistent with embodiments of the present disclosure. Masked image 300 may be a version of input image 200 such that masked region 302 represents a masked portion of input image 200. For example, editing object 202 in input image 200 may involve determining the masked region 302. In some embodiments, masked region 302 may be determined by selecting or outlining the area with a tool, such as eraser 110 in dashboard” Note: Ramesh teaches that an input image with a mask, making an image with a masked region, is input to its model where it can remove the object/element in the masked region.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Ramesh where a system that accepts an input image and mask and generates an intermediate image with the masked region filled in by inpainting generation, and passes the intermediate image to a second generation model which will output a synthetic image with higher quality will seek to remove an element that is masked in the intermediate image. There are several reasons that would motivate one to do so, Li’s two model approach allows for high quality inpainting to be done by first generating a coarse prediction that can then be increased in detail and texture by a second generator. Rather than recreating a masked region of an area many users may wish to instead remove an object that is not aesthetically pleasing, a common feature of photo editing software. This functionality could be implemented with the same high-quality results from Li’s two model approach if the intermediate representation sought to remove the element in its masked region rather than reproduce it before being refined by the second model. Regarding claim 17, Li teaches: A system comprising: a memory component; a processing device coupled to the memory component, (Li 5.1 “We developed our model using Pytorch 1.5.0 [22] and trained it on two NVIDIA GeForce RTX 2080Ti GPUs.” Note: Li teaches the use of a 2080ti GPU, a GPU with onboard memory teaching the presence of a processing device coupled to a memory component.) the processing device configured to perform operations comprising: obtaining an input image and an input mask, wherein the input image depicts an element and the input mask indicates a region of the element in the input image; (Li 3, cited in claim 1, teaches accepting an input image along with an input mask indicating an area to be generated over.) generating, using a first image generation model, an intermediate result based on the input image and the input mask, wherein the intermediate result includes first generated content of the element within the region indicated by the input mask; (Li 1, cited previously, teaches that the first stage of the model will generate an image where the masked area has its missing area generated by the model. As this produced image is only the output of G1 the coarse generator, and will be fed to the refinement generator later, Li teaches an intermediate result based on the input image and mask where the intermediate result modifies the masked region.) and generating, using a second image generation model, a synthetic image based on the input image and the intermediate result, wherein the synthetic image includes second generated content of the element within the region indicated by the input mask. (Li 4, cited in claim 1, teaches that after the intermediate result, the output of the coarse generator, is obtained, the refinement generate generates a final output image with a higher level of detail and texture) While Li teaches a two-model system where a first model accepts an input image with an input mask and seeks to generate content to fill the masked region it does not attempt to generate content in place of the masked element, instead aiming to accurately recreate the element. Editing an elements appearance or generating an entirely new object/element in its place when inpainting a masked region is detailed by Ramesh which teaches generating, using a first image generation model, an intermediate result based on the input image and the input mask, wherein the intermediate result includes first generated content in place of the element within the region indicated by the input mask; (Ramesh Col. 10 Line 24 “It will be appreciated that input images may include a variety of styles, including realistic images, photographs, digital art, oil paintings, and the like. Input image 200 may depict a variety of objects which may be edited, altered, or changed, such as plate object 202. For example, it may be desired to remove object 202, e dit various features of object 202 or replace object 202. FIG. 3 illustrates an example of a masked image 300, consistent with embodiments of the present disclosure. Masked image 300 may be a version of input image 200 such that masked region 302 represents a masked portion of input image 200. For example, editing object 202 in input image 200 may involve determining the masked region 302. In some embodiments, masked region 302 may be determined by selecting or outlining the area with a tool, such as eraser 110 in dashboard” Note: Ramesh teaches the object/element contained in the masked region can be replaced with new content entirely, or the existing object can have its features edited.) wherein the synthetic image includes second generated content in place of the element within the region indicated by the input mask (Ramesh Col. 10 line 24, cited above, teaches that a synthetic image generated by an image generation model can include new content in place of the previous content that was beneath the masked region.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Ramesh where a system that accepts an input image and mask and generates an intermediate image with the masked region filled in by inpainting generation, and passes the intermediate image to a second generation model which will output a synthetic image with higher quality will seek to remove an element that is masked in the intermediate image. There are several reasons that would motivate one to do so, a common feature in AI powered photo editing software is to replace an object with another that fits the image more, this feature many users take advantage of could be implemented to the described system if the model sought to replace content in a mask rather than recreate it . 07-21-aia AIA Claim s 4 is rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Li2 (InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following) . Regarding claim 4, Li teaches: The method of claim 1, further comprising: Li does not detail receiving a prompt detailing how its image generation of the masked region should occur. Determining image generation in response to a text prompt is taught by Li2 which teaches receiving a removal prompt, wherein the removal prompt comprises a command to remove an element from the input image; (Li2 Abstract “We propose InstructAny2Pix, a flexible multi-modal instruction-following system that enables users to edit an input image using instructions involving audio, images, and text ” 1 introduction “Concrete examples of such instructions can be ‘add the [sound] to [image],’ where the sound can be that of a dog barking or a piece of music. It can also be ‘add [object A] and remove [object B] from [image],’”) and selecting a removal mode based on the removal prompt, wherein the intermediate result is based on the removal mode. ( PNG media_image3.png 954 870 media_image3.png Greyscale Note: Previously it was shown that Li2 can accept text inputs which it will then read to recognize that a removal prompt was given. In Table 4 the templates, or modes, that the model recognizes are listed, teaching that a removal mode/template will be chosen in response to a removal prompt.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Li2 where inpainting a masked region of an input image accepts a corresponding text input that allows users to specify how generation should be handled, including the ability to specify that objects should be removed. There are several reasons that would motivate one to do so, object removal is a common feature of photo editing software, the model’s use could be further extended if users were allowed to provide a text prompt detailing which objects to remove, or alternatively edit or change. The ability for users to specify how generation should occur with a text prompt extends the models abilities and increases ease of use by providing a simple intuitive way to interact with the model . 07-21-aia AIA Claim s 5 is rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Xu (SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting) . Regarding claim 5, Li teaches: The method of claim 1, wherein: While Li teaches that the detail and texture equality of the intermediate result is enhanced by the refinement stage model it is not directly taught that the resolution of the image is enhanced. This is found in Xu which teaches the intermediate result comprises an intermediate image having a lower resolution than the synthetic image. (Xu 1 Introduction “To this end, we propose a novel deep learning framework for HR image inpainting. The framework mainly consists of two deep learning modules: (1) a low-resolution inpainting module for the reconstruction of high-frequency information in the missing region, and (2) a super-resolution module for the enhancement of the resolution of the inpainted region.” Note: Xu teaches a similar system to Li where a first deep learning model fills in an area of an input image covered by a mask. In Xu this intermediate result is taught to be low resolution, which the final synthetic image output by the super-resolution module will enhance to a higher resolution.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Xu where an intermediate result that has its texture and detail enhanced by a second generation model to produce the final output synthetic image also has its resolution enhanced to produce the synthetic image. There are several reasons that would motivate one to do so, as Li already seeks to enhance the quality of its intermediate result by improving detail and texture quality the quality could be further enhanced by increasing the resolution with known methods like super resolution models Xu leverages . 07-21-aia AIA Claim s 8 is rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Lugmayr (RePaint: Inpainting Using Denoising Diffusion Probabilistic Models) . Regarding claim 8, Li teaches: The method of claim 1, wherein generating the synthetic image comprises: Li does not however teach that a second noise input can be obtained and denoised to produce the final synthetic image. This is taught in Lugmayr which teaches obtaining a second noise input; and denoising the second noise input to generate the synthetic image. (Lugmayr Abstract “A Denoising Diffusion Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We employ a pretrained unconditional DDPM as the generative prior . To condition the generation process, we only alter the reverse diffusion iterations by sampling the unmasked regions using the given image information. Since this technique does not modify or condition the original DDPM network itself, the model produces high quality and diverse output images for any inpainting form.” 2 “we are leveraging on the high expressiveness of a pretrained Denoising Diffusion Probabilistic Model [12] (DDPM) and therefore use it as a prior for generic image inpainting .” 1 “In essence, the DDPM is trained to iteratively denoise the image by reversing a diffusion process. Starting from randomly sam pled noise, the DDPM is then iteratively applied for a certain number of steps, which yields the final image sample.” PNG media_image4.png 304 1352 media_image4.png Greyscale Note: Lugmayr teaches that a denoising model is responsible for making the generative prior, which as seen in Lugmayr 2 is an image that has been inpainted meaning the masked area of the input image has been filled by the DDPM model. After this first amount of noise is added and denoised the generative prior, or intermediate result, has been produced. The image is then passed into a DDPM again iteratively. As seen in by Lugmayr 1 and Fig. 3 the image iterates the reverse diffusion process over several cycles until the final output image that has had a second, or nth, amount of noise added and then denoised produces the final output image with a higher enhanced quality and accuracy.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Lugmayr where the second model that produces the final synthetic image is a diffusion model adding a second amount of noise after a first amount was added to produce the intermediate result. There are several reasons that would motivate one to do so, Li teaches a system where after a first image is generated its quality can be enhanced by passing the image to a model to produce the final synthetic output. High quality and realistic image generation, known benefits of diffusion models, could be leveraged by using a diffusion model as the second generator to produce an enhanced quality image . 07-21-aia AIA Claim s 10, 11, 14, 16, 18, are rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Ramesh (US 11983806 B1) and further in view of Suin (Distillation-guided Image Inpainting) . Regarding claims 10, Li teaches: The method of claim 1, wherein: While Li teaches elements can be recreated it does not teach that they can be removed. This is taught by Ramesh where the first image generation model is trained to remove an image element (Ramesh Col. 10 Line 24, cited previously, teaches that an object/element that is masked can be removed by the image generation model.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Ramesh where the inpainting a generative model performs seeks to remove an element in a designated area rather than simply recreating the missing area. There are several reasons that would motivate one to do so, a common feature in AI powered photo editing software is to remove an object for any number of reasons a user may desire, such as an object clashing with the photo’s overall aesthetic, by having the generation model remove an element rather than recreating it this functionality could be implemented. While Ramesh teaches that a model can be trained to remove an image element/object it does not teach a distillation learning technique that leverages a student/teacher model approach. This is found in Suin which teaches the first image generation model is trained to recreate an image element using a predicted image generated by a teacher image generation model. (Suin 1 “We deploy two networks: an auxiliary network (AN) and an inpainting network (IN), where both have a similar encoder-decoder backbone with three levels. The AN is used only for training to provide accurate information on what the missing regions should contain . We start with an under-complete auto encoder as our AN, which takes the ground truth image as input and tries to produce the same as output” PNG media_image5.png 634 1538 media_image5.png Greyscale Note: The specifications define the teacher image generation model as ¶7 “include obtaining a training set comprising an input image including an element, generating, using a teacher image generation model, a predicted image that replaces the element from the input image with generated content, and training, using the training set and the predicted image, a first image generation model to replace the element from the input image with the generated content.” This use of a teacher image generation model is clearly taught by Suin which similarly uses its teacher model, the AN, to generate a predicted image from the ground truth image. This knowledge of features learned from the predicted teacher image is distilled into the model that will inpaint the image, teaching an image generation model trained to recreate an image element that has been masked using a predicted image generated by a teacher model.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Suin where a model that is trained by using a predicted image of a teacher image generation model can train the model to remove an element rather than reconstructing the element. There are several reasons that would motivate one to do so, a common feature of photo editing tools is a remove feature that allows unwanted elements/objects of an image to be removed. Rather than simply recreating a masked area a model could be trained with Suin’s knowledge distillation student/teacher approach to accurately remove the object or element to extend the model’s functionality to solve a common problem many users wish to solve in photo editing. Regarding claim 14, Li teaches: The non-transitory computer readable medium of claim 12, wherein Other than the preamble of claim 14 which mentions the non-transitory computer readable medium, which was shown to be taught previously in the rejection of claim 12, the body of claim 14 is identical to claim 10 and is rejected under the same rationale. Regarding claim 11, Li teaches: The method of claim 10, wherein: While Li teaches that the element covered by a mask is recreated as accurately as possible it does not teach that the element should be replaced with new content the second image generation model is trained to replace the element from the input image based on an output of the first image generation model. (Suin 1, cited in claim 10, teaches that a teacher model, AN, generates a predicted image from a ground truth image and its knowledge is distilled into the actual functional model IN. The teacher AN model exists “for training to provide accurate information on what the missing regions should contain .” PNG media_image6.png 518 738 media_image6.png Greyscale Note: An example of AN’s function with IN can be seen above, where the results in the right most column show that the model after distillation from the teacher performs significantly at replacing the image elements than without.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Suin where a model that is trained by using a predicted image of a teacher image generation model can train the model to replace an element rather than reconstructing the element. There are several reasons that would motivate one to do so, a common feature of photo editing tools is a remove feature that allows unwanted elements/objects to change in appearance. Rather than simply recreating a masked area a model could be trained with Suin’s knowledge distillation student/teacher approach to accurately change the object or element to extend the model’s functionality to solve a common problem many users wish to solve in photo editing. Regarding claim 16, Li teaches: The non-transitory computer readable medium of claim 12, Other than the preamble of claim 16 which mentions the non-transitory computer readable medium, which was shown to be taught previously in the rejection of claim 12, the body of claim 16 is identical to claim 11 and is rejected under the same rationale. Regarding claim 18, Li teaches: The system of claim 17, further comprising: While Li teaches that a model can be trained to recreate masked portions of an image it does not teach that the model can remove content that has been masked, this is taught by Ramesh which teaches wherein the image generation model is trained to remove the element from the input image. ( Ramesh Col. 10 Line 24, cited previously, teaches that an object/element that is masked can be removed by the image generation model, as the model is a machine learning model it is implicit that some level of training was required for Ramesh’s model to obtain the present results.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Ramesh where the inpainting a generative model performs seeks to remove an element in a designated area rather than simply recreating the missing area. There are several reasons that would motivate one to do so, a common feature in AI powered photo editing software is to remove an object for any number of reasons a user may desire, such as an object clashing with the photo’s overall aesthetic, by having the generation model remove an element rather than recreating it this functionality could be implemented. While Ramesh teaches that a model may be trained to remove an element from the input image it does not teach that the training process can be aided by a teacher image generation model. Leveraging a teacher image generation model to train a model to generate content of a masked region is taught by Suin which teaches a teacher image generation model, wherein the teacher image generation model is trained to recreate the element from the input image. ( PNG media_image6.png 518 738 media_image6.png Greyscale Note: As seen in the AN, or teacher model, as taught by the previously cited Suin 1, accurately recreates a missing portion of an image.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Suin where a model that can remove an element contained in a masked region of an input image can be trained to do so via a teacher image generation model. There are several reasons that would motivate one to do so, a model may be accurately trained to recreate missing image parts, as the ground truth images for the complete original input are available it may be difficult to train a model to remove an object, as ground truth images without the object may not be available or difficult to gather. The training of the models training could be generalized further to train the model on a task such as element removal by leveraging a teacher/student training approach, a method known to increase generalization in a model . 07-21-aia AIA Claim s 13 is rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Ramesh (US 11983806 B1) and further in view of Li2 (InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following) . Regarding claim 13, Li teaches: The non-transitory computer readable medium of claim 12, the operations further comprising: Li does not detail receiving a prompt detailing how its image generation of the masked region should occur. Determining image generation in response to a text prompt is taught by Li2 which teaches receiving a removal prompt, wherein the removal prompt comprises a command to remove an element from the input image; (Li2 Abstract “We propose InstructAny2Pix, a flexible multi-modal instruction-following system that enables users to edit an input image using instructions involving audio, images, and text ” 1 introduction “Concrete examples of such instructions can be ‘add the [sound] to [image],’ where the sound can be that of a dog barking or a piece of music. It can also be ‘add [object A] and remove [object B] from [image],’”) and selecting a removal mode based on the removal prompt, wherein the intermediate result is based on the removal mode. ( PNG media_image3.png 954 870 media_image3.png Greyscale Note: Previously it was shown that Li2 can accept text inputs which it will then read to recognize that a removal prompt was given. In Table 4 the templates, or modes, that the model recognizes are listed, teaching that a removal mode/template will be chosen in response to a removal prompt.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Li2 where inpainting a masked region of an input image accepts a corresponding text input that allows users to specify how generation should be handled, including the ability to specify that objects should be removed. There are several reasons that would motivate one to do so, object removal is a common feature of photo editing software, the model’s use could be further extended if users were allowed to provide a text prompt detailing which objects to remove, or alternatively edit or change. The ability for users to specify how generation should occur with a text prompt extends the models abilities and increases ease of use by providing a simple intuitive way to interact with the model . 07-21-aia AIA Claim s 15, 19, are rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Ramesh (US 11983806 B1) and further in view of Lugmayr (RePaint: Inpainting Using Denoising Diffusion Probabilistic Models) . Regarding claim 15, Li teaches: The non-transitory computer readable medium of claim 12, While Li teaches that its model can be trained it does not teach training a diffusion model leveraging diffusion loss to do so, this is found in Lugmary which teaches wherein the first image generation model is trained by computing a diffusion loss and updating parameters of the first image generation model based on the diffusion loss. (Lugmayr 3 PNG media_image7.png 910 522 media_image7.png Greyscale Note: As seen in the Lugmayr excerpt, the model computes its loss using the equation (4) above. From this computer loss equation, the model is trained where it can parameterize the model, or in other words it can update/determine the parameters of the model based on the diffusion loss.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Lugmayr where a model has its parameters updated based on diffusion loss. There are several reasons that would motivate one to do so, previously in claim 8 it was shown why a diffusion model’s high-quality outputs would make a good fit for Li to leverage as its model. If a diffusion model is being leveraged then adjusting the model’s parameters based on diffusion loss is an efficient available method to minimize error/loss to obtain a more accurate model. Regarding claim 19, Li teaches: The system of claim 17, wherein: Li does not detail the use of diffusion models; this is found in Lugmayr which teaches the first image generation model and the second image generation model are diffusion models. (Lugmayr 2, and Fig. 3, cited previously, teach that the first image generation model equivalent that produces the intermediate result, the generative prior, is a DDPM. The final enhanced result, the synthetic image produced by the second image generation model is obtained from passing the generative prior through a DDPM once again, teaching that the first and second models can both be diffusion models.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Lugmayr where a system that involves a first model that produces an intermediate, lower quality representation passes its result to a second model that produces a higher quality final synthetic image uses diffusion models for both. There are several reasons that would motivate one to do so, as diffusion models are popular well-known models for producing high quality images the quality of the image at both stages could be improved by leveraging diffusion models . 07-21-aia AIA Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Li (DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting) in view of Ramesh (US 11983806 B1) and further in view of Xu (SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting) . Regarding claim 20, Li teaches: The system of claim 17, wherein: While Li teaches two separate image generation models, it does not teach that they have a different number of parameters. Xu teaches the first image generation model has fewer parameters than the second image generation model. ( PNG media_image8.png 806 1090 media_image8.png Greyscale Note: As seen in paragraph (1) above the first stage’s model has two regularization parameters, much less than are used in the second model in (3) which lists 8 regularization parameters, teaching that the second model which refines and enhances resolution of the first model’s intermediate result has more parameters.) It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Li with Xu where the model that produces the rougher, lower quality intermediate result has less parameters than the second model which refines the image into the final synthetic image. There are several reasons that would motivate one to do so, enhancing quality and resolution of an image to make a final product may be more demanding than simply generating a rough intermediate image. To efficiently balance resources, the first model could be made to be lighter by having less parameters than the second, heavier model with more parameters. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN GREGORY HAKALA whose telephone number is (571)272-7863. The examiner can normally be reached 8:00am-5:00pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /ALAN GREGORY HAKALA/Examiner, Art Unit 2617 /KING Y POON/ Supervisory Patent Examiner, Art Unit 2617 Application/Control Number: 18/956,284 Page 2 Art Unit: 2617 Application/Control Number: 18/956,284 Page 3 Art Unit: 2617 Application/Control Number: 18/956,284 Page 4 Art Unit: 2617 Application/Control Number: 18/956,284 Page 5 Art Unit: 2617 Application/Control Number: 18/956,284 Page 6 Art Unit: 2617 Application/Control Number: 18/956,284 Page 7 Art Unit: 2617 Application/Control Number: 18/956,284 Page 8 Art Unit: 2617 Application/Control Number: 18/956,284 Page 9 Art Unit: 2617 Application/Control Number: 18/956,284 Page 10 Art Unit: 2617 Application/Control Number: 18/956,284 Page 11 Art Unit: 2617 Application/Control Number: 18/956,284 Page 12 Art Unit: 2617 Application/Control Number: 18/956,284 Page 13 Art Unit: 2617 Application/Control Number: 18/956,284 Page 14 Art Unit: 2617 Application/Control Number: 18/956,284 Page 15 Art Unit: 2617 Application/Control Number: 18/956,284 Page 16 Art Unit: 2617 Application/Control Number: 18/956,284 Page 17 Art Unit: 2617 Application/Control Number: 18/956,284 Page 18 Art Unit: 2617 Application/Control Number: 18/956,284 Page 19 Art Unit: 2617 Application/Control Number: 18/956,284 Page 20 Art Unit: 2617 Application/Control Number: 18/956,284 Page 21 Art Unit: 2617 Application/Control Number: 18/956,284 Page 22 Art Unit: 2617 Application/Control Number: 18/956,284 Page 23 Art Unit: 2617 Application/Control Number: 18/956,284 Page 24 Art Unit: 2617 Application/Control Number: 18/956,284 Page 25 Art Unit: 2617 Application/Control Number: 18/956,284 Page 26 Art Unit: 2617 Application/Control Number: 18/956,284 Page 27 Art Unit: 2617 Application/Control Number: 18/956,284 Page 28 Art Unit: 2617 Application/Control Number: 18/956,284 Page 29 Art Unit: 2617
Prosecution Insights