DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed July 1, 2026 has been entered. Claims 1-20 remain pending in the application. Applicant’s amendments to the Specification and Claims have overcome the objections and 101 rejections previously set forth in the Non-Final Office Action mailed April 1, 2026.
Response to Arguments
Applicant's arguments filed July 1, 2026, regarding Wang’s lack of teaching for the limitation, “modifying, by one or more trained latent diffusion models (LDMs), the reference image based upon the determined features and the received input, wherein any one or more of a background of the reference image, an area of the reference image or a style of the reference image are modified” have been fully considered but they are not persuasive. The examiner respectfully disagrees with Applicant's assertion that Wang does not teach and/or disclose “modifying, by one or more trained latent diffusion models (LDMs), the reference image based upon the determined features and the received input, wherein any one or more of a background of the reference image, an area of the reference image or a style of the reference image are modified”. Wang teaches modifying, by one or more trained latent diffusion models (LDMs), the reference image based upon the determined features and the received input, wherein any one or more of a background of the reference image, an area of the reference image or a style of the reference image are modified (Fig. 1 Caption Page 1, Fig. 2 Caption on Page 2, Paragraph 3 in 2nd Col. of Page 3 – “Given an image, a user defined mask, and a text prompt, Imagen Editor makes localized edits to the designated areas. The model meaningfully incorporates the user’s intent and performs photorealistic edits… All of the diffusion models, i.e., the base model and super-resolution (SR) models, condition on high-resolution 1024×1024 image and mask inputs…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”). Referencing the quote, the given image is the reference image, which the Imagen Editor makes localized edits to. These localized edits are modifications that are based on diffusion latents (features), along with a mask and text prompt (input). Imagen Editor is a latent diffusion model (Paragraph 1 and 3 in 2nd Col. of Page 3 – “Imagen Editor is a diffusion-based model [6] fine-tuned from Imagen [36] for editing…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”). An example of the modifications made by Imagen Editor to an area of an image is shown in Fig. 1 of Wang.
PNG
media_image1.png
183
652
media_image1.png
Greyscale
Screenshot of Fig. 1 (taken from Wang)
Therefore, Wang does teach that a reference image is modified by a latent diffusion model based on features and input.
On the other hand, Applicant’s arguments, filed July 1, 2026, regarding Wang’s lack of teaching for the limitation, “determining, by the one or more LDMs, whether a spatial output of the modified image satisfies predetermined criteria; and causing to display, by the user interface of the service, the modified image based on the determining the spatial output of the modified image satisfies the predetermined criteria” have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Rando et al. (Red-Teaming the Stable Diffusion Safety Filter).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting) in view of Rando et al. (Red-Teaming the Stable Diffusion Safety Filter), hereinafter Wang and Rando respectively.
Regarding claim 1, Wang teaches a method (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor is a text-guided image inpainting model targeting improved representation and reflection of linguistic inputs, fine-grained control and high fidelity outputs”) comprising:
receiving, by a user interface of a service, a reference image and an input comprising text associated with the reference image (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor takes three inputs from the user, 1) the image to be edited, 2) a binary mask to specify the edit region, and 3) a text prompt – and all three inputs are used to guide the output samples”; Note: Imagen Editor provides the service, and a user interface is implied because the user would not be able to provide input without an interface), wherein the reference image comprises any one or more of an object or a person (Fig. 18 – The figure shows that the input/reference image has an object or person (or both) in it; see screenshot of Fig. 18 below);
PNG
media_image2.png
371
1004
media_image2.png
Greyscale
Screenshot of Fig. 18 (taken from Wang)
determining, by a trained machine learning (ML) model, one or more features of the reference image (Paragraph 2 in 2nd Col. of Page 1, Paragraph 3 in 1st Col. of Page 2, Paragraph 3 in 2nd Col. of Page 3 – “Imagen Editor adds image and mask context to each diffusion stage via three convolutional downsampling image encoders… In terms of text-image alignment, Imagen Editor trained with object-masking is preferred…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the diffusion latents are the features, and they are derived using convolutional image encoders. Convolutional image encoders are a type of machine learning model. Imagen Editor, which is made-up of convolutional image encoders (see Fig. 2 below), is trained);
modifying, by one or more trained latent diffusion models (LDMs), the reference image based upon the determined features and the received input, wherein any one or more of a background of the reference image, an area of the reference image or a style of the reference image are modified (Fig. 1 Caption Page 1, Fig. 2 Caption on Page 2, Paragraph 3 in 1st Col. of Page 2, Paragraph 3 in 2nd Col. of Page 3 – “Given an image, a user defined mask, and a text prompt, Imagen Editor makes localized edits to the designated areas. The model meaningfully incorporates the user’s intent and performs photorealistic edits… All of the diffusion models, i.e., the base model and super-resolution (SR) models, condition on high-resolution 1024×1024 image and mask inputs… In terms of text-image alignment, Imagen Editor trained with object-masking is preferred…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the mask area of the reference/input image is modified via Imagen Editor, which is trained and made-up of latent diffusion models (see Fig. 2 below). The modifications are based on the inputs and the diffusion latents. The diffusion latents are the features);
and causing to display, by the user interface of the service, the modified image (Fig. 1 – The figure below shows the output modified image. It would have been obvious to display it to the user because the user requested the image by providing the input and thus would want to see the output).
PNG
media_image3.png
241
976
media_image3.png
Greyscale
Screenshot of Fig. 1 (taken from Wang)
PNG
media_image4.png
275
496
media_image4.png
Greyscale
Screenshot of Fig. 2 (taken from Wang)
Wang does not teach determining, by the one or more LDMs, whether a spatial output of the modified image satisfies predetermined criteria; and causing to display, by the user interface of the service, the modified image based on the determining the spatial output of the modified image satisfies the predetermined criteria. However, Rando teaches determining, by the one or more LDMs, whether a spatial output of the modified image satisfies predetermined criteria (Paragraph 5 on Page 2 – “Here is a simplified outline for the safety filter in Stable Diffusion v1.4 (see Figure 1, and Appendix C for the pseudocode): The user provides a prompt, say “a photograph of an astronaut riding a horse”. The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed. Each of these fixed vectors represents some pre-defined sensitive concepts. Every concept has a prespecified similarity threshold. If the cosine similarity between the image and any of the concepts is larger than the respective threshold, the image is discarded”; Note: the stable diffusion model, which is an LDM, determines whether the image embedding in latent space, which is the spatial output, satisfies a predetermined criteria. The criteria is whether the similarity between the image embedding in latent space and predefined concepts is less than a similarity threshold. The modified image was previously taught by Wang and is represented by the created image in this case); and causing to display, by the user interface of the service, the modified image based on the determining the spatial output of the modified image satisfies the predetermined criteria (Fig. 1 Caption on Page 1, Paragraph 5 on Page 2 – “Simplified safety filter algorithm implemented in Stable Diffusion v1.4. Images are mapped to a CLIP latent space, where they are compared against pre-computed embeddings of 17 unsafe concepts (see full list in Appendix E). If the cosine similarity between the output image and any of the concepts is above a certain threshold, the image is considered unsafe and blacked-out…The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed”; Note: the image is only displayed if it was determined that the criteria of the similarity threshold was satisfied (similarity between image embedding in latent space and predefined concepts is less than the similarity threshold). The displaying step is shown in Fig. 1; see below). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rando to determine whether a spatial output satisfies predetermined criteria and display the modified image based on the determining because “Due to possible safety risks, Stable Diffusion does include a post-hoc safety filter that blocks explicit images…Teams that deploy popular models (whether in open-source or closed-source) should have a formal security policy and a dedicated contact for responsible disclosure” (Rando: Paragraph 2 on Page 2, Paragraph 3 on Page 5). In other words, the predetermined criteria help with providing a layer of safety and security in the image modification/generation by preventing the display of inappropriate content.
PNG
media_image5.png
477
924
media_image5.png
Greyscale
Screenshot of Fig. 1 (taken from Rando)
Regarding claim 2, Wang in view of Rando teaches the method of claim 1. Wang further teaches wherein the input comprises a mask of an area in the image (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor takes three inputs from the user, 1) the image to be edited, 2) a binary mask to specify the edit region, and 3) a text prompt – and all three inputs are used to guide the output samples”).
Regarding claim 6, Wang in view of Rando teaches the method of claim 1. Wang does not teach assessing, by a privacy LDM, whether any one or more of the reference image, the received input or the modified image meets the predetermined criteria prior to the reference image being displayed on the user interface of the service, wherein the predetermined criteria comprises any one or more of violence, profanity or nudity. However, Rando teaches assessing, by a privacy LDM, whether any one or more of the reference image, the received input or the modified image meets the predetermined criteria prior to the reference image being displayed on the user interface of the service (Paragraph 5 on Page 2 – “Here is a simplified outline for the safety filter in Stable Diffusion v1.4 (see Figure 1, and Appendix C for the pseudocode): The user provides a prompt, say “a photograph of an astronaut riding a horse”. The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed. Each of these fixed vectors represents some pre-defined sensitive concepts. Every concept has a prespecified similarity threshold. If the cosine similarity between the image and any of the concepts is larger than the respective threshold, the image is discarded”; Note: the stable diffusion model, which is the privacy LDM, determines whether the image embedding in latent space, which represents the modified image in this case, meets a predetermined criteria before displaying, as shown in Fig. 1 above. The criteria is whether the similarity between the image embedding in latent space and predefined concepts are less than a similarity threshold), wherein the predetermined criteria comprises any one or more of violence, profanity or nudity (Paragraph 6 on Page 2 – “Because CLIP is trained to match the embeddings of images and corresponding textual captions, it is expected that the textual embedding of some unsafe concept (e.g., “nudity”) will be close to the image embeddings of depictions of this same concept. The 17 text prompts that were used to produce pre-computed CLIP embeddings fully determine the unsafe concepts that Stable Diffusion’s safety filter looks for”; Note: the predetermined criteria use concepts, such as nudity). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rando to assess the input on whether or not it meets a predetermined criteria of violence, profanity, or nudity because “Due to possible safety risks, Stable Diffusion does include a post-hoc safety filter that blocks explicit images…Teams that deploy popular models (whether in open-source or closed-source) should have a formal security policy and a dedicated contact for responsible disclosure” (Rando: Paragraph 2 on Page 2, Paragraph 3 on Page 5). In other words, the predetermined criteria help with providing a layer of safety and security in the image modification/generation by preventing the display of inappropriate content and ensuring proper usage of the software.
Claims 3, 7-10, 13-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Rando and Karpman et al. (US 11995803 B1), hereinafter Karpman.
Regarding claim 3, Wang in view of Rando teaches the method of claim 1. Wang does not teach transmitting prior to receiving the input, by the user interface, any one or more of a suggestion of a mask of the area in the image or a suggestion affecting the background or the style of the reference image. However, Karpman teaches transmitting prior to receiving the input, by the user interface, any one or more of a suggestion of a mask of the area in the image or a suggestion affecting the background or the style of the reference image (Fig. 4A, Col. 20 lines 56-63 – “The generation interface 400 also includes a style menu 404 that enables the user to browse and select among pre-set image styles for the image generation request. In the example of FIG. 4A, the style menu 404 displays a set (e.g., array) of style option tiles, each style option tile including a text description of the image style (e.g., anime, Van Gogh, oil painting, line drawing, digital art, etc.) and a sample image in the corresponding style”; Note: the user interface has suggestions for image style. The reference image was previously taught by Wang in the rejection of claim 1. Fig. 4A below shows that the style suggestions are populated prior to receiving input).
PNG
media_image6.png
506
441
media_image6.png
Greyscale
Screenshot of Fig. 4A (taken from Karpman)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to provide suggestions of a style because “the software application layer 124 and inspiration affordance can assist the user in deciding on a type of image they wish to generate by providing visual examples of (possible) image styles and enabling users to select among pre-set styles without requiring the user to develop and/or phrase image generation prompts that include stylistic constraints, thereby reducing the cognitive burden on the user in operating the software application layer and increasing the likelihood that the image subsequently generated by the text-to-image diffusion model 112 will align with user expectations” (Karpman: Col. 21 lines 17-27). In other words, the suggestions can make it easier for the user to make decisions regarding the image edits and generation.
Regarding claim 7, Wang in view of Rando teaches the method of claim 6. Wang does not teach transmitting, by the user interface based upon the assessment, a request to update any one or more of the reference image or the received input; and receiving, by the user interface, any one or more of an updated reference image or an updated input. However, Karpman teaches transmitting, by the user interface based upon the assessment, a request to update any one or more of the reference image or the received input (Col. 23 lines 6-23 – “In implementations where image generation tasks are subject to content moderation, the software application layer 124 is configured to transition display of the waiting screen back to the generation interface (e.g., the generation interface shown in FIG. 4A) in response to receiving the error message from the communication interface 122 (e.g., based on outputs of the text classifier, based on outputs of the visual classifier). In these implementations, the software application layer 124 can display the error message (e.g., as a pop-up dialog box over the generation interface or adjacent to the interactive text field) informing the user that system declined to generate the requested image (e.g., “Could not generate that image—please try again with a different prompt”). The error message displayed to the user may also include a reason for the moderation decision based on a classification of the prompt by the text classifier or based on a classification of the base image by the visual classifier”); and receiving, by the user interface, any one or more of an updated reference image or an updated input (Col. 23 lines 31-35 and 39-47 – “Results interface 414 displays (a preview of) the image(s) 1 to 6 generated by the text-to-image diffusion model 112 at 416 in response to the image generation request and a description field 418 that displays the text prompt submitted to the model…The results interface 414 may also include a regenerate affordance (e.g., a button, a link) that enables the user to submit another image generation request…More specifically, in response to receiving a user input (e.g., a touch input to a display, a click) over the description field, the software application can allow the user to edit text displayed in the description field (e.g., by displaying a touch keyboard below the description field, through keyboard inputs)”; Note: the user can re-enter a new input, and in response, generate an image). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to re-prompt the user and receive an updated user input for the benefit of giving the user a chance to change their input and use the software in the appropriate way. If the software did not allow for updating input, then it may be too restrictive and prevent users from using the service.
Regarding claim 8, Wang teaches a system (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor is a text-guided image inpainting model targeting improved representation and reflection of linguistic inputs, fine-grained control and high fidelity outputs”; Note: Imagen Editor is the system) comprising the instructions of:
receiving, by a user interface of a service, a reference image and an input comprising text associated with the reference image (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor takes three inputs from the user, 1) the image to be edited, 2) a binary mask to specify the edit region, and 3) a text prompt – and all three inputs are used to guide the output samples”; Note: Imagen Editor provides the service, and a user interface is implied because the user would not be able to provide input without an interface);
determining, by a trained machine learning (ML) model, one or more features of the reference image (Paragraph 2 in 2nd Col. of Page 1, Paragraph 3 in 1st Col. of Page 2, Paragraph 3 in 2nd Col. of Page 3 – “Imagen Editor adds image and mask context to each diffusion stage via three convolutional downsampling image encoders… In terms of text-image alignment, Imagen Editor trained with object-masking is preferred…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the diffusion latents are the features, and they are derived using convolutional image encoders. Convolutional image encoders are a type of machine learning model. Imagen Editor, which is made-up of convolutional image encoders (see Fig. 2 above), is trained);
modifying, by one or more trained latent diffusion models (LDMs), the reference image based upon the determined features and the received input, wherein any one or more of a background of the reference image, an area of the reference image or a style of the reference image are modified (Fig. 1 Caption Page 1, Fig. 2 Caption on Page 2, Paragraph 3 in 1st Col. of Page 2, Paragraph 3 in 2nd Col. of Page 3 – “Given an image, a user defined mask, and a text prompt, Imagen Editor makes localized edits to the designated areas. The model meaningfully incorporates the user’s intent and performs photorealistic edits… All of the diffusion models, i.e., the base model and super-resolution (SR) models, condition on high-resolution 1024×1024 image and mask inputs… In terms of text-image alignment, Imagen Editor trained with object-masking is preferred…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the mask area of the reference/input image is modified via Imagen Editor, which is trained and made-up of latent diffusion models (see Fig. 2 above). The modifications are based on the inputs and the diffusion latents. The diffusion latents are the features);
and causing to display, by the user interface of the service, the modified image (Fig. 1 – The figure shows the output modified image. It would have been obvious to display it to the user because the user requested the image by providing the input and thus would want to see the output).
Wang does not teach a non-transitory memory with instructions stored thereon; and a processor operably coupled to the non-transitory memory and configured to execute the instructions. However, Karpman teaches a non-transitory memory with instructions stored thereon; and a processor operably coupled to the non-transitory memory and configured to execute the instructions (Col. 24 lines 26-33 – “Storage device 505 may include a non-transitory computer-readable storage medium holding information, instructions, or some combination thereof, for example instructions that when executed by the processor 501, cause processor 501 to be configured or operable to perform one or more operations of a method as described herein”; Note: it is implied that the processor and non-transitory memory are coupled since the processor would not be able to access or execute the instructions otherwise). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to have a non-transitory memory with instructions coupled to a processor to execute the instructions because the diffusion models in Wang would not have been able to run properly without a processor and would not exist without code to define the models. Additionally, a non-transitory memory would provide a persistent and reliable storage for future use of the instructions and data.
Furthermore, Wang modified by Karpman does not teach determining, by the one or more LDMs, whether a spatial output of the modified image satisfies predetermined criteria; and causing to display, by the user interface of the service, the modified image based on the determining the spatial output of the modified image satisfies the predetermined criteria. However, Rando teaches determining, by the one or more LDMs, whether a spatial output of the modified image satisfies predetermined criteria (Paragraph 5 on Page 2 – “Here is a simplified outline for the safety filter in Stable Diffusion v1.4 (see Figure 1, and Appendix C for the pseudocode): The user provides a prompt, say “a photograph of an astronaut riding a horse”. The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed. Each of these fixed vectors represents some pre-defined sensitive concepts. Every concept has a prespecified similarity threshold. If the cosine similarity between the image and any of the concepts is larger than the respective threshold, the image is discarded”; Note: the stable diffusion model, which is an LDM, determines whether the image embedding in latent space, which is the spatial output, satisfies a predetermined criteria. The criteria is whether the similarity between the image embedding in latent space and predefined concepts is less than a similarity threshold. The modified image was previously taught by Wang and is represented by the created image in this case); and causing to display, by the user interface of the service, the modified image based on the determining the spatial output of the modified image satisfies the predetermined criteria (Fig. 1 Caption on Page 1, Paragraph 5 on Page 2 – “Simplified safety filter algorithm implemented in Stable Diffusion v1.4. Images are mapped to a CLIP latent space, where they are compared against pre-computed embeddings of 17 unsafe concepts (see full list in Appendix E). If the cosine similarity between the output image and any of the concepts is above a certain threshold, the image is considered unsafe and blacked-out…The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed”; Note: the image is only displayed if it was determined that the criteria of the similarity threshold was satisfied (similarity between image embedding in latent space and predefined concepts is less than the similarity threshold). The displaying step is shown in Fig. 1; see above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rando to determine whether a spatial output satisfies predetermined criteria and display the modified image based on the determining because “Due to possible safety risks, Stable Diffusion does include a post-hoc safety filter that blocks explicit images…Teams that deploy popular models (whether in open-source or closed-source) should have a formal security policy and a dedicated contact for responsible disclosure” (Rando: Paragraph 2 on Page 2, Paragraph 3 on Page 5). In other words, the predetermined criteria help with providing a layer of safety and security in the image modification/generation by preventing the display of inappropriate content.
Regarding claim 9, Wang in view of Rando and Karpman teaches the system of claim 8. Wang further teaches wherein the input comprises a mask of an area in the image (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor takes three inputs from the user, 1) the image to be edited, 2) a binary mask to specify the edit region, and 3) a text prompt – and all three inputs are used to guide the output samples”).
Regarding claim 10, Wang in view of Rando and Karpman teaches the system of claim 8. Wang does not teach wherein the processor when further configured to execute the instructions of: transmitting prior to receiving the input, by the user interface, any one or more of a suggestion of a mask of the area in the image or a suggestion affecting the background or style of the reference image. However, Karpman teaches transmitting prior to receiving the input, by the user interface, any one or more of a suggestion of a mask of the area in the image or a suggestion affecting the background or the style of the reference image (Fig. 4A, Col. 20 lines 56-63 – “The generation interface 400 also includes a style menu 404 that enables the user to browse and select among pre-set image styles for the image generation request. In the example of FIG. 4A, the style menu 404 displays a set (e.g., array) of style option tiles, each style option tile including a text description of the image style (e.g., anime, Van Gogh, oil painting, line drawing, digital art, etc.) and a sample image in the corresponding style”; Note: the user interface has suggestions for image style. The reference image was previously taught by Wang in the rejection of claim 8. Fig. 4A above shows that the style suggestions are populated prior to receiving input). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to provide suggestions of a style because “the software application layer 124 and inspiration affordance can assist the user in deciding on a type of image they wish to generate by providing visual examples of (possible) image styles and enabling users to select among pre-set styles without requiring the user to develop and/or phrase image generation prompts that include stylistic constraints, thereby reducing the cognitive burden on the user in operating the software application layer and increasing the likelihood that the image subsequently generated by the text-to-image diffusion model 112 will align with user expectations” (Karpman: Col. 21 lines 17-27). In other words, the suggestions can make it easier for the user to make decisions regarding the image edits and generation.
Regarding claim 13, Wang in view of Rando and Karpman teaches the system of claim 8. Wang does not teach wherein the processor when further configured to execute the instructions of: assessing, by a privacy LDM, whether any one or more of the reference image, the received input or the modified image meets the predetermined criteria prior to the reference image being displayed on the user interface of the service. However, Rando teaches assessing, by a privacy LDM, whether any one or more of the reference image, the received input or the modified image meets the predetermined criteria prior to the reference image being displayed on the user interface of the service (Paragraph 5 on Page 2 – “Here is a simplified outline for the safety filter in Stable Diffusion v1.4 (see Figure 1, and Appendix C for the pseudocode): The user provides a prompt, say “a photograph of an astronaut riding a horse”. The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed. Each of these fixed vectors represents some pre-defined sensitive concepts. Every concept has a prespecified similarity threshold. If the cosine similarity between the image and any of the concepts is larger than the respective threshold, the image is discarded”; Note: the stable diffusion model, which is the privacy LDM, determines whether the image embedding in latent space, which represents the modified image in this case, meets a predetermined criteria before displaying, as shown in Fig. 1 above. The criteria is whether the similarity between the image embedding in latent space and predefined concepts are less than a similarity threshold). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rando to assess the input on whether or not it meets a predetermined criteria because “Due to possible safety risks, Stable Diffusion does include a post-hoc safety filter that blocks explicit images…Teams that deploy popular models (whether in open-source or closed-source) should have a formal security policy and a dedicated contact for responsible disclosure” (Rando: Paragraph 2 on Page 2, Paragraph 3 on Page 5). In other words, the predetermined criteria help with providing a layer of safety and security in the image modification/generation by preventing the display of inappropriate content and ensuring proper usage of the software.
Regarding claim 14, Wang in view of Rando and Karpman teaches the system of claim 13. Wang does not teach wherein the predetermined criteria comprises any one or more of violence, profanity or nudity. However, Rando teaches wherein the predetermined criteria comprises any one or more of violence, profanity or nudity (Paragraph 6 on Page 2 – “Because CLIP is trained to match the embeddings of images and corresponding textual captions, it is expected that the textual embedding of some unsafe concept (e.g., “nudity”) will be close to the image embeddings of depictions of this same concept. The 17 text prompts that were used to produce pre-computed CLIP embeddings fully determine the unsafe concepts that Stable Diffusion’s safety filter looks for”; Note: the predetermined criteria use concepts, such as nudity). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rando to have the predetermined criteria comprise violence, profanity, or nudity because they are considered “unsafe concepts” (Rando: Paragraph 6 on Page 2), and “Due to possible safety risks, Stable Diffusion does include a post-hoc safety filter that blocks explicit images…Teams that deploy popular models (whether in open-source or closed-source) should have a formal security policy and a dedicated contact for responsible disclosure” (Rando: Paragraph 2 on Page 2, Paragraph 3 on Page 5). In other words, the predetermined criteria help with providing a layer of safety and security in the image modification/generation by preventing the display of what is considered an unsafe concept and thus, ensuring proper usage of the software.
Regarding claim 15, Wang in view of Rando and Karpman teaches the system of claim 13. Wang does not teach wherein the processor when further configured to execute the instructions of: transmitting, by the user interface based upon the assessment, a request to update any one or more of the reference image or the received input; and receiving, by the user interface, any one or more of an updated reference image or an updated input. However, Karpman teaches transmitting, by the user interface based upon the assessment, a request to update any one or more of the reference image or the received input (Col. 23 lines 6-23 – “In implementations where image generation tasks are subject to content moderation, the software application layer 124 is configured to transition display of the waiting screen back to the generation interface (e.g., the generation interface shown in FIG. 4A) in response to receiving the error message from the communication interface 122 (e.g., based on outputs of the text classifier, based on outputs of the visual classifier). In these implementations, the software application layer 124 can display the error message (e.g., as a pop-up dialog box over the generation interface or adjacent to the interactive text field) informing the user that system declined to generate the requested image (e.g., “Could not generate that image—please try again with a different prompt”). The error message displayed to the user may also include a reason for the moderation decision based on a classification of the prompt by the text classifier or based on a classification of the base image by the visual classifier”); and receiving, by the user interface, any one or more of an updated reference image or an updated input (Col. 23 lines 31-35 and 39-47 – “Results interface 414 displays (a preview of) the image(s) 1 to 6 generated by the text-to-image diffusion model 112 at 416 in response to the image generation request and a description field 418 that displays the text prompt submitted to the model…The results interface 414 may also include a regenerate affordance (e.g., a button, a link) that enables the user to submit another image generation request…More specifically, in response to receiving a user input (e.g., a touch input to a display, a click) over the description field, the software application can allow the user to edit text displayed in the description field (e.g., by displaying a touch keyboard below the description field, through keyboard inputs)”; Note: the user can re-enter a new input, and in response, generate an image). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to re-prompt the user and receive an updated user input for the benefit of giving the user a chance to change their input and use the software in the appropriate way. If the software did not allow for updating input, then it may be too restrictive and prevent users from using the service.
Regarding claim 16, Wang teaches program instructions (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor is a text-guided image inpainting model targeting improved representation and reflection of linguistic inputs, fine-grained control and high fidelity outputs”; Note: Imagen Editor is comprised of program instructions) that cause:
receiving, by a user interface of a service, a reference image and an input comprising text associated with the reference image (Paragraph 1 in 2nd Col. of Page 3 – “Imagen Editor takes three inputs from the user, 1) the image to be edited, 2) a binary mask to specify the edit region, and 3) a text prompt – and all three inputs are used to guide the output samples”; Note: Imagen Editor provides the service, and a user interface is implied because the user would not be able to provide input without an interface), wherein the reference image comprises any one or more of an object or a person (Fig. 18 – The figure shows that the input/reference image has an object or person (or both) in it; see screenshot of Fig. 18 above);
determining, by a trained machine learning (ML) model, one or more features of the reference image (Paragraph 2 in 2nd Col. of Page 1, Paragraph 3 in 1st Col. of Page 2, Paragraph 3 in 2nd Col. of Page 3 – “Imagen Editor adds image and mask context to each diffusion stage via three convolutional downsampling image encoders… In terms of text-image alignment, Imagen Editor trained with object-masking is preferred…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the diffusion latents are the features, and they are derived using convolutional image encoders. Convolutional image encoders are a type of machine learning model. Imagen Editor, which is made-up of convolutional image encoders (see Fig. 2 above), is trained);
modifying, by one or more trained latent diffusion models (LDMs), the reference image based upon the determined features and the received input, wherein any one or more of a background of the reference image, an area of the reference image or a style of the reference image are modified (Fig. 1 Caption Page 1, Fig. 2 Caption on Page 2, Paragraph 3 in 1st Col. of Page 2, Paragraph 3 in 2nd Col. of Page 3 – “Given an image, a user defined mask, and a text prompt, Imagen Editor makes localized edits to the designated areas. The model meaningfully incorporates the user’s intent and performs photorealistic edits… All of the diffusion models, i.e., the base model and super-resolution (SR) models, condition on high-resolution 1024×1024 image and mask inputs… In terms of text-image alignment, Imagen Editor trained with object-masking is preferred…In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the mask area of the reference/input image is modified via Imagen Editor, which is trained and made-up of latent diffusion models (see Fig. 2 above). The modifications are based on the inputs and the diffusion latents. The diffusion latents are the features);
and causing to display, by the user interface of the service, the modified image (Fig. 1 – The figure shows the output modified image. It would have been obvious to display it to the user because the user requested the image by providing the input and thus would want to see the output).
Wang does not teach a non-transitory computer readable medium comprising program instructions stored thereon which when executed by a processor effectuate the method. However, Karpman teaches a non-transitory computer readable medium comprising program instructions stored thereon which when executed by a processor effectuate the method (Col. 24 lines 26-33 – “Storage device 505 may include a non-transitory computer-readable storage medium holding information, instructions, or some combination thereof, for example instructions that when executed by the processor 501, cause processor 501 to be configured or operable to perform one or more operations of a method as described herein”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to have a non-transitory computer readable medium with instructions coupled to a processor to execute the instructions because the diffusion models in Wang would not have been able to run properly without a processor and would not exist without code to define the models. Additionally, a non-transitory memory would provide a persistent and reliable storage for future use of the instructions and data.
Furthermore, Wang modified by Karpman does not teach determining, by the one or more LDMs, whether a spatial output of the modified image satisfies predetermined criteria; and causing to display, by the user interface of the service, the modified image based on the determining the spatial output of the modified image satisfies the predetermined criteria. However, Rando teaches determining, by the one or more LDMs, whether a spatial output of the modified image satisfies predetermined criteria (Paragraph 5 on Page 2 – “Here is a simplified outline for the safety filter in Stable Diffusion v1.4 (see Figure 1, and Appendix C for the pseudocode): The user provides a prompt, say “a photograph of an astronaut riding a horse”. The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed. Each of these fixed vectors represents some pre-defined sensitive concepts. Every concept has a prespecified similarity threshold. If the cosine similarity between the image and any of the concepts is larger than the respective threshold, the image is discarded”; Note: the stable diffusion model, which is an LDM, determines whether the image embedding in latent space, which is the spatial output, satisfies a predetermined criteria. The criteria is whether the similarity between the image embedding in latent space and predefined concepts is less than a similarity threshold. The modified image was previously taught by Wang and is represented by the created image in this case); and causing to display, by the user interface of the service, the modified image based on the determining the spatial output of the modified image satisfies the predetermined criteria (Fig. 1 Caption on Page 1, Paragraph 5 on Page 2 – “Simplified safety filter algorithm implemented in Stable Diffusion v1.4. Images are mapped to a CLIP latent space, where they are compared against pre-computed embeddings of 17 unsafe concepts (see full list in Appendix E). If the cosine similarity between the output image and any of the concepts is above a certain threshold, the image is considered unsafe and blacked-out…The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed”; Note: the image is only displayed if it was determined that the criteria of the similarity threshold was satisfied (similarity between image embedding in latent space and predefined concepts is less than the similarity threshold). The displaying step is shown in Fig. 1; see above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rando to determine whether a spatial output satisfies predetermined criteria and display the modified image based on the determining because “Due to possible safety risks, Stable Diffusion does include a post-hoc safety filter that blocks explicit images…Teams that deploy popular models (whether in open-source or closed-source) should have a formal security policy and a dedicated contact for responsible disclosure” (Rando: Paragraph 2 on Page 2, Paragraph 3 on Page 5). In other words, the predetermined criteria help with providing a layer of safety and security in the image modification/generation by preventing the display of inappropriate content.
Regarding claim 17, Wang in view of Rando and Karpman teaches the computer readable medium of claim 16. Wang does not teach wherein the program instructions which when executed by the processor further effectuate: transmitting prior to receiving the input, by the user interface, any one or more of a suggestion of a mask of the area in the image or a suggestion affecting the background or the style of the reference image. However, Karpman teaches transmitting prior to receiving the input, by the user interface, any one or more of a suggestion of a mask of the area in the image or a suggestion affecting the background or the style of the reference image (Fig. 4A, Col. 20 lines 56-63 – “The generation interface 400 also includes a style menu 404 that enables the user to browse and select among pre-set image styles for the image generation request. In the example of FIG. 4A, the style menu 404 displays a set (e.g., array) of style option tiles, each style option tile including a text description of the image style (e.g., anime, Van Gogh, oil painting, line drawing, digital art, etc.) and a sample image in the corresponding style”; Note: the user interface has suggestions for image style. The reference image was previously taught by Wang in the rejection of claim 16. Fig. 4A above shows that the style suggestions are populated prior to receiving input). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to provide suggestions of a style because “the software application layer 124 and inspiration affordance can assist the user in deciding on a type of image they wish to generate by providing visual examples of (possible) image styles and enabling users to select among pre-set styles without requiring the user to develop and/or phrase image generation prompts that include stylistic constraints, thereby reducing the cognitive burden on the user in operating the software application layer and increasing the likelihood that the image subsequently generated by the text-to-image diffusion model 112 will align with user expectations” (Karpman: Col. 21 lines 17-27). In other words, the suggestions can make it easier for the user to make decisions regarding the image edits and generation.
Regarding claim 19, Wang in view of Rando and Karpman teaches the computer readable medium of claim 16. Wang does not teach wherein the program instructions which when executed by the processor further effectuate: assessing, by a privacy LDM, whether any one or more of the reference image, the received input or the modified image meets the predetermined criteria prior to the reference image being displayed on the user interface of the service, wherein the predetermined criteria comprises any one or more of violence, profanity or nudity. However, Rando teaches assessing, by a privacy LDM, whether any one or more of the reference image, the received input or the modified image meets the predetermined criteria prior to the reference image being displayed on the user interface of the service (Paragraph 5 on Page 2 – “Here is a simplified outline for the safety filter in Stable Diffusion v1.4 (see Figure 1, and Appendix C for the pseudocode): The user provides a prompt, say “a photograph of an astronaut riding a horse”. The Stable Diffusion model then creates an image conditioned on this prompt. Before being shown to the user, the image is run through CLIP’s image encoder [13] to obtain an embedding; i.e., a high-dimensional vector representation of the input. Then, the cosine similarity between this embedding and 17 different fixed embedding vectors is computed. Each of these fixed vectors represents some pre-defined sensitive concepts. Every concept has a prespecified similarity threshold. If the cosine similarity between the image and any of the concepts is larger than the respective threshold, the image is discarded”; Note: the stable diffusion model, which is the privacy LDM, determines whether the image embedding in latent space, which represents the modified image in this case, meets a predetermined criteria before displaying, as shown in Fig. 1 above. The criteria is whether the similarity between the image embedding in latent space and predefined concepts are less than a similarity threshold), wherein the predetermined criteria comprises any one or more of violence, profanity or nudity (Paragraph 6 on Page 2 – “Because CLIP is trained to match the embeddings of images and corresponding textual captions, it is expected that the textual embedding of some unsafe concept (e.g., “nudity”) will be close to the image embeddings of depictions of this same concept. The 17 text prompts that were used to produce pre-computed CLIP embeddings fully determine the unsafe concepts that Stable Diffusion’s safety filter looks for”; Note: the predetermined criteria use concepts, such as nudity). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rando to assess the input on whether or not it meets a predetermined criteria of violence, profanity, or nudity because “Due to possible safety risks, Stable Diffusion does include a post-hoc safety filter that blocks explicit images…Teams that deploy popular models (whether in open-source or closed-source) should have a formal security policy and a dedicated contact for responsible disclosure” (Rando: Paragraph 2 on Page 2, Paragraph 3 on Page 5). In other words, the predetermined criteria help with providing a layer of safety and security in the image modification/generation by preventing the display of inappropriate content and ensuring proper usage of the software.
Regarding claim 20, Wang in view of Rando and Karpman teaches the computer readable medium of claim 16. Wang does not teach wherein the program instructions which when executed by the processor further effectuate: transmitting, by the user interface based upon the assessment, a request to update any one or more of the reference image or the received input; and receiving, by the user interface, any one or more of an updated reference image or an updated input. However, Karpman teaches transmitting, by the user interface based upon the assessment, a request to update any one or more of the reference image or the received input (Col. 23 lines 6-23 – “In implementations where image generation tasks are subject to content moderation, the software application layer 124 is configured to transition display of the waiting screen back to the generation interface (e.g., the generation interface shown in FIG. 4A) in response to receiving the error message from the communication interface 122 (e.g., based on outputs of the text classifier, based on outputs of the visual classifier). In these implementations, the software application layer 124 can display the error message (e.g., as a pop-up dialog box over the generation interface or adjacent to the interactive text field) informing the user that system declined to generate the requested image (e.g., “Could not generate that image—please try again with a different prompt”). The error message displayed to the user may also include a reason for the moderation decision based on a classification of the prompt by the text classifier or based on a classification of the base image by the visual classifier”); and receiving, by the user interface, any one or more of an updated reference image or an updated input (Col. 23 lines 31-35 and 39-47 – “Results interface 414 displays (a preview of) the image(s) 1 to 6 generated by the text-to-image diffusion model 112 at 416 in response to the image generation request and a description field 418 that displays the text prompt submitted to the model…The results interface 414 may also include a regenerate affordance (e.g., a button, a link) that enables the user to submit another image generation request…More specifically, in response to receiving a user input (e.g., a touch input to a display, a click) over the description field, the software application can allow the user to edit text displayed in the description field (e.g., by displaying a touch keyboard below the description field, through keyboard inputs)”; Note: the user can re-enter a new input, and in response, generate an image). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Karpman to re-prompt the user and receive an updated user input for the benefit of giving the user a chance to change their input and use the software in the appropriate way. If the software did not allow for updating input, then it may be too restrictive and prevent users from using the service.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Rando and Liu et al. (DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection), hereinafter Liu.
Regarding claim 4, Wang in view of Rando teaches the method of claim 1. Wang does not teach wherein an identity of the object or the person is preserved. However, Liu teaches wherein an identity of the object or the person is preserved (Paragraph 3 in 1st Col. of Page 2 – “we propose DiffProtect, which utilizes a pre trained diffusion autoencoder [37] to generate adversarial images for facial privacy protection. The overall pipeline of DiffProtect is shown in Fig. 2. We first encode an input face image I into a high-level semantic code z and a low-level noise code xT. We then iteratively optimize an adversarial semantic code zadv such that the resulting protected image Ip generated by the conditional DDIM decoding process [37, 50] can fool the face recognition model”; Note: the identity of the person is preserved). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Liu to preserve the identity of the person because “the widely deployed FR [face recognition] systems also pose a huge threat to personal privacy as billions of users have publicly shared their photos on social media. Through large-scale social media photo analysis, FR systems can be used for detecting user relationships [46], stalking victims [47], stealing identities [32], and performing massive government surveillance [42, 17, 34]. It is urgent to develop facial privacy protection techniques to protect individuals from unauthorized FR systems” (Liu: Paragraph 2 in 1st Col. of Page 1). In other words, preserving a person’s identify provides a layer of safety and privacy for the person.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Rando and Kim et al. (Zoom-to-Inpaint: Image Inpainting with High-Frequency Details), hereinafter Kim.
Regarding claim 5, Wang in view of Rando teaches the method of claim 1. Wang further teaches detecting whether a resolution of the reference image exceeds a capacity of the LDMs (Paragraph 4 in 2nd Col. of Page 3, Paragraph 1 in 1st Col. of Page 4 – “The conditioning image and the corresponding mask input to Imagen Editor are always at 1024×1024 resolution. The base diffusion 64×64 model and the 64×64→256×256 super-resolution model operate at a smaller resolution, and thus require some form of downsampling to match the diffusion latent resolution”; Note: the conditioning/reference image is detected to have a higher resolution than the diffusion model); determining a section of the reference image being preserved (Paragraph 4 in 2nd Col. of Page 3 – “In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the mask is a section of the image being preserved); and editing a section of the reference image being preserved and pasting the edited section into the reference image (Fig. 1 Caption on Page 1, Fig. 2 Caption on Page 2 – “Given an image, a user defined mask, and a text prompt, Imagen Editor makes localized edits to the designated areas… All of the diffusion models, i.e., the base model and super-resolution (SR) models, condition on high-resolution 1024×1024 image and mask inputs”; Note: Fig. 2 above shows that the mask area of the image are upscaled and edited. Fig. 1 above shows the edit being pasted onto the input/reference image (the spacesuit and headphones are pasted onto the image of the dog)). Wang does not teach that the edited section is zoomed-in, from the limitation: “editing a zoomed-in section of the reference image being preserved and pasting the edited section into the reference image”. However, Kim teaches editing a zoomed-in section of the reference image being preserved (Fig. 5, Paragraph 2 in 2nd Col. of Page 4 – “our proposed refinement is achieved by zooming in, refining, then zooming out back to the input resolution”; Note: Fig. 5 shows the zoomed-in section of the image based on the mask and how that section is edited to restore missing regions). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Kim to zoom-in on the mask because “This allows the refinement network to correct local irregularities at a finer level and to learn from high-resolution (HR) labels, thus effectively reducing the spectral bias at the desired resolution and injecting more high-frequency details into the resulting image” (Kim: paragraph 2 in 1st Col. of Page 2). In other words, it would help produce a better-quality output.
PNG
media_image7.png
798
566
media_image7.png
Greyscale
Screenshot of Fig. 5 (taken from Kim)
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Rando, Karpman, and Liu.
Regarding claim 11, Wang in view of Rando and Karpman teaches the system of claim 8. Wang further teaches wherein: the reference image comprises any one or more of an object or a person (Fig. 18 – The figure shows that the input/reference image has an object or person (or both) in it; see screenshot of Fig. 18 above). Wang does not teach an identity of the object or the person is preserved. However, Liu teaches wherein an identity of the object or the person is preserved (Paragraph 3 in 1st Col. of Page 2 – “we propose DiffProtect, which utilizes a pre trained diffusion autoencoder [37] to generate adversarial images for facial privacy protection. The overall pipeline of DiffProtect is shown in Fig. 2. We first encode an input face image I into a high-level semantic code z and a low-level noise code xT. We then iteratively optimize an adversarial semantic code zadv such that the resulting protected image Ip generated by the conditional DDIM decoding process [37, 50] can fool the face recognition model”; Note: the identity of the person is preserved). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Liu to preserve the identity of the person because “the widely deployed FR [face recognition] systems also pose a huge threat to personal privacy as billions of users have publicly shared their photos on social media. Through large-scale social media photo analysis, FR systems can be used for detecting user relationships [46], stalking victims [47], stealing identities [32], and performing massive government surveillance [42, 17, 34]. It is urgent to develop facial privacy protection techniques to protect individuals from unauthorized FR systems” (Liu: Paragraph 2 in 1st Col. of Page 1). In other words, preserving a person’s identify provides a layer of safety and privacy for the person.
Claims 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Rando, Karpman, and Kim.
Regarding claim 12, Wang in view of Rando and Karpman teaches the system of claim 8. Wang further teaches wherein the processor when further configured to execute the instructions of: detecting whether a resolution of the reference image exceeds a capacity of the LDMs (Paragraph 4 in 2nd Col. of Page 3, Paragraph 1 in 1st Col. of Page 4 – “The conditioning image and the corresponding mask input to Imagen Editor are always at 1024×1024 resolution. The base diffusion 64×64 model and the 64×64→256×256 super-resolution model operate at a smaller resolution, and thus require some form of downsampling to match the diffusion latent resolution”; Note: the conditioning/reference image is detected to have a higher resolution than the diffusion model); determining a section of the reference image being preserved (Paragraph 4 in 2nd Col. of Page 3 – “In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the mask is a section of the image being preserved); and editing a section of the reference image being preserved and pasting the edited section into the reference image (Fig. 1 Caption on Page 1, Fig. 2 Caption on Page 2 – “Given an image, a user defined mask, and a text prompt, Imagen Editor makes localized edits to the designated areas… All of the diffusion models, i.e., the base model and super-resolution (SR) models, condition on high-resolution 1024×1024 image and mask inputs”; Note: Fig. 2 above shows that the mask area of the image are upscaled and edited. Fig. 1 above shows the edit being pasted onto the input/reference image (the spacesuit and headphones are pasted onto the image of the dog)). Wang does not teach that the edited section is zoomed-in, from the limitation: “editing a zoomed-in section of the reference image being preserved and pasting the edited section into the reference image”. However, Kim teaches editing a zoomed-in section of the reference image being preserved (Fig. 5, Paragraph 2 in 2nd Col. of Page 4 – “our proposed refinement is achieved by zooming in, refining, then zooming out back to the input resolution”; Note: Fig. 5 shows the zoomed-in section of the image based on the mask and how that section is edited to restore missing regions). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Kim to zoom-in on the mask because “This allows the refinement network to correct local irregularities at a finer level and to learn from high-resolution (HR) labels, thus effectively reducing the spectral bias at the desired resolution and injecting more high-frequency details into the resulting image” (Kim: paragraph 2 in 1st Col. of Page 2). In other words, it would help produce a better-quality output.
Regarding claim 18, Wang in view of Rando and Karpman teaches the computer readable medium of claim 16. Wang does not teach wherein the program instructions which when executed by the processor further effectuate: detecting whether a resolution of the reference image exceeds a capacity of the LDMs; determining a section of the reference image being preserved; and editing a zoomed-in section of the reference image being preserved and pasting the edited section into the reference image. Wang further teaches detecting whether a resolution of the reference image exceeds a capacity of the LDMs (Paragraph 4 in 2nd Col. of Page 3, Paragraph 1 in 1st Col. of Page 4 – “The conditioning image and the corresponding mask input to Imagen Editor are always at 1024×1024 resolution. The base diffusion 64×64 model and the 64×64→256×256 super-resolution model operate at a smaller resolution, and thus require some form of downsampling to match the diffusion latent resolution”; Note: the conditioning/reference image is detected to have a higher resolution than the diffusion model); determining a section of the reference image being preserved (Paragraph 4 in 2nd Col. of Page 3 – “In Imagen Editor, we modify Imagen to condition on both the image and the mask by concatenating them with the diffusion latents along the channel dimension”; Note: the mask is a section of the image being preserved); and editing a section of the reference image being preserved and pasting the edited section into the reference image (Fig. 1 Caption on Page 1, Fig. 2 Caption on Page 2 – “Given an image, a user defined mask, and a text prompt, Imagen Editor makes localized edits to the designated areas… All of the diffusion models, i.e., the base model and super-resolution (SR) models, condition on high-resolution 1024×1024 image and mask inputs”; Note: Fig. 2 above shows that the mask area of the image are upscaled and edited. Fig. 1 above shows the edit being pasted onto the input/reference image (the spacesuit and headphones are pasted onto the image of the dog)). Wang does not teach that the edited section is zoomed-in, from the limitation: “editing a zoomed-in section of the reference image being preserved and pasting the edited section into the reference image”. However, Kim teaches editing a zoomed-in section of the reference image being preserved (Fig. 5, Paragraph 2 in 2nd Col. of Page 4 – “our proposed refinement is achieved by zooming in, refining, then zooming out back to the input resolution”; Note: Fig. 5 shows the zoomed-in section of the image based on the mask and how that section is edited to restore missing regions). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Kim to zoom-in on the mask because “This allows the refinement network to correct local irregularities at a finer level and to learn from high-resolution (HR) labels, thus effectively reducing the spectral bias at the desired resolution and injecting more high-frequency details into the resulting image” (Kim: paragraph 2 in 1st Col. of Page 2). In other words, it would help produce a better-quality output.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kim et al. (Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Models) teaches a method of preventing harmful content generation in text-to-image diffusion models.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/ Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
