DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 12 objected to because of the following informalities:
Claim 12 Line 2 reciting “generating a training data element a target image” should be “generating a training data element, a target image”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fei et al. (“Supervised GAN Watermarking for Intellectual Property Protection”)(Hereinafter referred to as Fei) in view of Steins (“Stable Diffusion - The Invisible Watermark in Generated Images”).
Regarding Claim 1, Fei discloses A method for watermarking a GAN model, comprising: (Abstract, “We propose a watermarking method for protecting the Intellectual Property (IP) of Generative Adversarial Networks (GANs).”)
generating one or more training data elements, the one or more training data elements including target images and the target images including pre-defined watermark information; and (See Page 2 Right Column Paragraph 2, “This method works as follows: first, a given invisible watermark is embedded into the training data by exploiting a pre-trained network for image watermarking, then the watermarked data are used to train the GAN” In this case, a given invisible watermark corresponds to “pre-defined watermark information” and embedding a watermark into training data to get watermarked data corresponds to “generating one or more training data elements”. See Page 3 Fig. 1 showing the watermarking method.
Additionally see Page 1 Right Column Paragraph 4, “IPR protection is achieved by training the GAN in a supervised manner so that any generated image contains a prescribed invisible watermark.” Note that supervised training means that there is target data or “ground truth” labels being used.
Lastly see Page 3 Left Column Paragraph 2, “where xi denotes the input image, xw,i is the watermarked image, obtained at the output of the encoder, and n is the number of training images. The first term in (1) aims to minimize the distortion between x and xw, while the second term aims to minimize the error between the prescribed watermark and the watermark predicted by Dw.” Here Fei teaching a loss function to minimize the error between input image and watermarked image. This means that the watermarked data is considered as being a part of “target images” for this method as they would be a part of the ground truth that the model is being trained to generate.)
training the GAN model to predict the target images using training data including the one or more training data elements. (See Page 2 Right Column Paragraph 2, “This method works as follows: first, a given invisible watermark is embedded into the training data by exploiting a pre-trained network for image watermarking, then the watermarked data are used to train the GAN” Here Fei teaches training the model on the watermarked data, which are a part of the target images. Note that training the model to “predict the target images using training data” would obviously be implied as these are image generating models.)
However, Fei fails to explicitly disclose A method for watermarking a diffusion model, comprising:
training the diffusion model to predict the target images using training data including the one or more training data elements.
Steins teaches A method for watermarking a diffusion model, comprising: (See Pages 1-5 describing the idea of embedding an invisible watermark for stable diffusion generated images. Stable Diffusion being a well-known text to image diffusion model.)
training the diffusion model to predict the target images using training data including the one or more training data elements. (In combination with Fei which teaches training the model with the target images that has watermarked data, and Steins teaching a diffusion model, the above limitation is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Fei with Steins to include using the watermarking method on diffusion based models.
The motivation to combine Fei with Steins would have been obvious as both are in the field of applying watermarks to generative models. Steins is simply teaching the well-known concept of watermarks for diffusion models. See Steins Page 3 reciting, “While everyone is using Stable Diffusion to generate artwork, have you ever realized there is a watermark in the generated images?”.
Regarding Claim 18, Fei in view of Steins disclose A system comprising: a memory storing instructions; and at least one processor coupled to the memory, the processor being configured to execute the instructions to: (See Fei Abstract, “We propose a watermarking method for protecting the Intellectual Property (IP) of Generative Adversarial Networks (GANs).”Although not explicitly mentioned, it is well known and common that machine learning models are ran on computing devices which inherently have memory and processors.)
generate one or more training data elements, the one or more training data elements including target images and the target images including pre-defined watermark information; and train a diffusion model to predict the target images using training data including the one or more training data elements. (The above limitations are similar to those of Claim 1 and are therefore rejection under a similar rationale as those of Claim 1).
Regarding Claim 20, Fei in view of Steins disclose A non-transitory computer-readable medium comprising program instructions, which, when executed by one or more processors, cause the one or more processors to: (See Fei Abstract, “We propose a watermarking method for protecting the Intellectual Property (IP) of Generative Adversarial Networks (GANs).”Although not explicitly mentioned, it is well known and common that machine learning models are ran on computing devices which inherently have memory (non-transitory computer-readable medium) and processors.)
generate one or more training data elements, the one or more training data elements including target images and the target images including pre-defined watermark information; and train a diffusion model to predict the target images using training data including the one or more training data elements. (The above limitations are similar to those of Claim 1 and are therefore rejection under a similar rationale as those of Claim 1).
Claims 2-17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Fei in view of Steins and in further view of Michael (“Diffusion Models : Unconditional&Conditional Image Generation”).
Regarding Claim 2, Fei in view of Steins fails to explicitly disclose The method of claim 1, wherein the diffusion model is an unconditional diffusion model or a class-conditioned diffusion model.
Michael teaches wherein the diffusion model is an unconditional diffusion model or a class-conditioned diffusion model. (See Pages 1-2 describing unconditional diffusion models. See Pages 6-7 describing conditional diffusion models including specifically class conditional diffusion models.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Fei in view of Steins with Michael to include specifically having the diffusion model be either an unconditional diffusion model or a class-conditional diffusion model.
The motivation to combine would Fei in view of Steins with Michael have been obvious as Michael is simply describing well-known types of diffusion models. The benefit of using a unconditional diffusion model would be that no supervision is required (See Michael Page 1 Paragraph 2) and the benefit of a class-conditional model is that the image generation can be guided by class (See Michael Page 6 Paragraph 1).
Regarding Claim 3, Fei in view of Steins and Michael disclose The method of claim 1, further comprising: training the diffusion model using the training data to predict each of the target images from a corresponding noisy version of the target images. (See Michael Page 2 Paragraph 1, “These models learn to reverse a Markov chain that transforms the data into white Gaussian noise by training a neural network to predict the mean and covariance of a sequence of Gaussian distributions.” Also see Michael Page 3 Paragraph 1, “Instead of directly modeling the output at each step as in [1], DDPM proposes to learn the reverse process by estimating the noise in the image at each step. This modification results in an objective that resembles denoising score matching. The authors use the Pixel-CNN++ architecture, to predict the noise in an image.” Here Michael very briefly discusses the how diffusion models work.
Although Michael does not fully and explicitly disclose/explain the processes of diffusion, these processes are already well-known. Adding noise to an image (forward diffusion), and then learning to reverse this process to reconstruct the image from the noisy version (reverse diffusion). Specifically the reverse diffusion process can thus can be considered as “using the training data to predict each of the target images from a corresponding noisy version of the target images”. The motivation to combine would have been similar to that of Claim 2 rejection motivation.)
Regarding Claim 4, Fei in view of Steins and Michael disclose The method of claim 1, further comprising: generating the target images of the one or more training data elements by embedding the pre-defined watermark information into one or more original training images. (See Fei Page 2 Right Column Paragraph 2, “This method works as follows: first, a given invisible watermark is embedded into the training data by exploiting a pre-trained network for image watermarking, then the watermarked data are used to train the GAN” See Page 3 Fig. 1 showing the watermarking method and how it is embedded into the one or more original training images.)
Regarding Claim 5, Fei in view of Steins and Michael disclose The method of claim 4, further comprising: embedding the pre-defined watermark information into the one or more original training images by encoding the pre-defined watermark information by an encoder and (See Fei Page 2 Right Column Section “A. Watermarking Network” reciting, “The watermarking network is described by an encoder-decoder architecture. We denote the encoder and decoder network as Ew and Dw respectively. The output of Ew is the watermarked image, while the output of Dw is the watermark message.”
Also see Page 3 Fig. 1 showing the watermarking method with an encoder encoding the watermark information.)
including the encoded pre-defined watermark information into the one or more original training images. (See Fei Page 2 Right Column Paragraph 2, “then the watermarked data are used to train the GAN” Here Fei teaches training the model on the watermarked data, which are a part of the training data and thus can be considered as “including the encoded pre-defined watermark information into the one or more original training images.”)
Regarding Claim 6, Fei in view of Steins and Michael disclose The method of claim 1, wherein the pre-defined watermark information is an encoded binary string. (See Fei Page 3 Fig. 1 showing the watermark being a bit string.)
Regarding Claim 7, Fei in view of Steins and Michael disclose The method of claim 1, further comprising: verifying the diffusion model has been watermarked by generating an image by the diffusion model and checking whether the generated image contains pre-defined watermark information. (See Fei Abstract, “The aim is to watermark the GAN model so that any image generated by the GAN contains an invisible watermark (signature), whose presence inside the image can be checked at a later stage for ownership verification.”
Additionally see Fei Page 1 Left Column Paragraph 1, “Watermarking has been applied for digital media ownership verification by embedding digital watermarks into the cover media to be protected. The owner is able to prove media ownership by extracting the watermark from it.” Lastly see Fei Page 3 Fig. 1 showing the watermarking process. Note that it would be obvious to someone ordinarily skilled in the art to verify that the diffusion model has been watermarked by generating an image and verifying that it contains the watermark information.)
Regarding Claim 8, Fei in view of Steins and Michael disclose The method of claim 1, further comprising: determining whether another diffusion model corresponds to the diffusion model by generating an image by the diffusion model and determining whether the generated image contains pre-defined watermark information. (See Fei Abstract and Page 1 Left Column Paragraph 1 teaching to verify ownership by using the watermark. Lastly see Fei Page 3 Fig. 1 showing the watermarking process. In the case where there is another diffusion model, it would be obvious to apply the methodology taught by Fei to generating an image containing watermark information and verifying ownership of the model by comparing extracted watermarks to see if it matches the owner’s watermark.)
Regarding Claim 9, Fei in view of Steins and Michael disclose The method of claim 8, further comprising: determining whether the generated image contains pre-defined watermark information by a watermark decoder trained to extract the pre-defined watermark information from generated images. (See Fei Page 3 Fig. 1 showing a trained decoder used to extract a predicted watermark (predefined watermark).)
Regarding Claim 10, Fei in view of Steins and Michael disclose The method of claim 1, wherein the diffusion model is a text-to-image generation model. (See Steins Pages 1-5 describing the idea of embedding an invisible watermark for stable diffusion generated images. Stable Diffusion being a well-known text to image diffusion model. The motivation to combine would have been similar to that of Claim 2 rejection motivation.)
Regarding Claim 11, Fei in view of Steins and Michael disclose The method of claim 1, wherein at least one of the target images is a pre-defined watermark image. (See Fei Page 2 Right Column Paragraph 2, “This method works as follows: first, a given invisible watermark is embedded into the training data by exploiting a pre-trained network for image watermarking, then the watermarked data are used to train the GAN”. Note that in this case, the watermarked data which was generated by embedding an invisible watermark into an image corresponds to a “pre-defined watermark image”. Additionally see Fei Page 1 Right Column Paragraph 4 teaching supervised training. Since it’s supervised, that means that the watermarked data can be considered to be “one of the target images”.)
Regarding Claim 12, Fei in view of Steins and Michael disclose The method of claim 1, further comprising: generating a training data element a target image, the target image being a pre-defined watermark image. (See Fei Page 2 Right Column Paragraph 2, “This method works as follows: first, a given invisible watermark is embedded into the training data by exploiting a pre-trained network for image watermarking, then the watermarked data are used to train the GAN”. Note that in this case, the watermarked data which was generated by embedding an invisible watermark into an image corresponds to a “pre-defined watermark image”. Additionally see Fei Page 1 Right Column Paragraph 4 teaching supervised training. Since it’s supervised, that means that the watermarked data can be considered as “the target image being a pre-defined watermark image”.)
Regarding Claim 13, Fei in view of Steins and Michael disclose The method of claim 12, wherein the training data element is an image-text pair comprising the target image and a text prompt for the diffusion model. (See Michael Page 14 Paragraph 2, “Previous methods for training text-to-image generation models require a large number of high-quality image-text pairs.” The motivation to combine would have been similar to that of Claim 2 rejection motivation.)
Regarding Claim 14, Fei in view of Steins and Michael disclose The method of claim 13, further comprising: training the diffusion model to predict the target image from the text prompt. (See Steins Pages 1-5 describing the idea of embedding an invisible watermark for stable diffusion generated images. Stable diffusion is a well-known trained diffusion model used to predict the target image from the text prompt. The motivation to combine would have been similar to that of Claim 2 rejection motivation.)
Regarding Claim 15, Fei in view of Steins and Michael disclose The method of claim 13, further comprising: verifying that the diffusion model has been watermarked by determining whether the diffusion model generates the target image from the text prompt. (See Fei Abstract, “The aim is to watermark the GAN model so that any image generated by the GAN contains an invisible watermark (signature), whose presence inside the image can be checked at a later stage for ownership verification.”
Additionally see Fei Page 1 Left Column Paragraph 1, “Watermarking has been applied for digital media ownership verification by embedding digital watermarks into the cover media to be protected. The owner is able to prove media ownership by extracting the watermark from it.” See Fei Page 3 Fig. 1 showing the watermarking process.
Lastly see Steins Pages 1-5 describing the idea of embedding an invisible watermark for stable diffusion generated images. Stable Diffusion being a well-known text to image diffusion model.
Note that it would be obvious to someone ordinarily skilled in the art to verify that the a text to image based diffusion model has been watermarked by generating an image from a text prompt and verifying that it contains the watermark information.)
Regarding Claim 16, Fei in view of Steins and Michael disclose The method of claim 13, further comprising: determining whether a second diffusion model corresponds to the diffusion model by checking whether the second diffusion model generates the target image from the text prompt. (See Fei Abstract and Page 1 Left Column Paragraph 1 teaching to verify ownership by using the watermark. Lastly see Fei Page 3 Fig. 1 showing the watermarking process.
Lastly see Steins Pages 1-5 describing Stable Diffusion, a well-known text to image diffusion model. In the case where there is second diffusion model, it would be obvious to apply the methodology taught by Fei to generating an image from a text which should contain watermark information and verifying ownership of the model by comparing extracted watermarks to see if it matches the owner’s watermark.)
Regarding Claim 17, Fei in view of Steins and Michael disclose The method of claim 1, further comprising: training the diffusion model using supervised training using the target images as ground truth. (See Fei Page 1 Right Column Paragraph 4, “IPR protection is achieved by training the GAN in a supervised manner so that any generated image contains a prescribed invisible watermark.” Note that supervised training means that there is target data or “ground truth” labels being used.
Lastly see Fei Page 3 Left Column Paragraph 2, “where xi denotes the input image, xw,i is the watermarked image, obtained at the output of the encoder, and n is the number of training images. The first term in (1) aims to minimize the distortion between x and xw, while the second term aims to minimize the error between the prescribed watermark and the watermark predicted by Dw.” Here Fei teaching a loss function to minimize the error between input image and watermarked image. This means that the watermarked data is considered as being a part of “target images” for this method as they would be a part of the ground truth that the model is being trained to generate.)
Regarding Claim 19, Claim 19 contains similar limitations as to Claim 2 and is therefore rejected under a similar rationale as Claim 2.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611