DETAIL OFFICE ACTIONS
The United States Patent & Trademark Office appreciates the response filed for the current application that is submitted on 03/30/2026. The United States Patent & Trademark Office reviewed the following documents submitted and has made the following comments below.
Amendment
Applicant submitted amendments on 03/30/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Applicant Arguments:
Applicant/s state/s that the cited prior arts do not teach the limitation “transforming, using a diffusion model, a text condition to a second latent representation compatible with the latent representation of the identity”; therefore, the rejection under 35 U.S.C. 103 should be withdrawn.
Examiner’s Responses:
In response to the Applicant’s argument, filed 03/30/2026, The Examiner respectfully disagrees. The Examiner finds that Liu teaches a diffusion model to modify a selfie image of user based on user’s text prompt. Specifically, the diffusion model consists of an image encoder and a text encoder. The image encoder analyzes input image and turns the user’s facial or body features into embeddings, which are compact numerical representations. The text encoder converts user input text and those embeddings into a feature vector that the diffusion model can use. The diffusion model generates a synthesized image that keeps the person’s identity while changing style, clothing, pose, background, or theme.
The Applicant argues that Liu does not teaches the limitation “transforming, using a diffusion model, a text condition to a second latent representation compatible with the latent representation of the identity”. The Examiner states that the Applicant is reading only the quotation and missing the cited paragraph as well as the mentioned structure. For aforementioned limitation, the Examiner cited ¶ 0034-0035 and figure 2 of Liu (Office Action mailed 12/31/2025, page 4-5). In the cited materials, Liu discloses a trained machine learning diffusion model, using a text encoder – a component of said diffusion model, to receive a user input text and transform the user input text into feature vector. (See annotated ¶ 0035 and FIG. 2 below).
PNG
media_image1.png
806
1786
media_image1.png
Greyscale
PNG
media_image2.png
1706
3045
media_image2.png
Greyscale
The Applicant mistakenly considers the text encoder and the diffusion model as two different, separate structures, and falsely assume that the diffusion model is not used to transform the “text condition”; when in fact, the text encoder is a sub component of the diffusion model. Therefore, the Examiner finds that the limitation reads on Liu. Furthermore, The Examiner made a proper determination of obviousness under 35 U.S.C. §103, and also provided an appropriate supporting rationale in view of the recent decision by the Supreme Court in KSR International Co. v. Teleflex Inc. (KSR), 550 U.S. 398, 82 USPQ2d 1385 (2007). The Examiner’s rational are based on the Office’s current understanding of the law, and are believed to be fully consistent with the binding precedent of the Supreme Court. Furthermore, the Examiner supported the rejection under 35 U.S.C. §103 via making the clear articulation of the reason(s) why the claimed invention would have been obvious by citing the specific areas in the prior art references. Last, the Examiner, clearly stating the modification of the inventions, supported the rejection under 35 U.S.C. §103 by making the analysis explicit. Last, the Examiner did not make conclusory statements. The Court quoting In re Kahn, 441 F.3d 977, 988, 78 USPQ2d 1329, 1336 (Fed. Cir. 2006), stated that “‘[R]ejections on obviousness cannot be sustained by mere conclusory statements; instead, there must be some articulated reasoning with some rational underpinning to support the legal conclusion of obviousness.’” KSR, 550 U.S. at ___, 82 USPQ2d at 1396. Therefore, the Examiner has established a proper 35 U.S.C. §103 rejection with Liu in view of Zhi-Song, in view of Ghosh, which is disclosed in detail below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-8, 10-13, 15-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over
Liu et al. (US-20240282016-A1, filed 02/21/2023, hereinafter Liu) in view of
Zhi-Song Liu et al. (Liu, Zhi-Song, Wan-Chi Siu, and Yui-Lam Chan. "Reference based face super-resolution." IEEE access 7 (2019): 129112-129126.), and further in view of
Ghosh et al. (US-20240404225-A1, filed 07/05/2023, hereinafter Ghosh).
CLAIM 1
In regards to Claim 1, Liu teaches a method, comprising: receiving an input image, the input image comprising a selfie (Liu, ¶ [0019-0020]: “capturing an image of a user (referred to as input image … position at least some part of the body of the user (e.g., the face) at a designated location in a field of view of the camera)”; see input image 500 and 600 in figures 5 and 6, respectively); transforming, using a neural network (Liu, ¶ [0028-0029]: “the image encoder includes a vision transformer (ViT)”), the input image to a latent representation (Liu, ¶ [0028-0029]: “The image encoder is configured to receive the image of the user and generate a set of embeddings that semantically describes visual features of the user based at least on the image of the user. The set of embeddings is selected from a lexicon of embeddings within a latent space…” Liu teaches using a vision transformer to transform input image into a set of embeddings) of an identity (Liu, ¶ [0032]: “the set of embeddings generated for the image of the user is associated with a user identifier”, ¶ [0053]: “different user identifiers… determined from the different input images”. Liu teaches generating transforming input images to embeddings that describe facial features of user, associated with an identifier, different input will generate different identifier); transforming, using a diffusion model (Liu, ¶ [0035]: “the trained machine learning diffusion model ”), a text condition (Liu, ¶ [0035]: “The user input text may include style and/or scene words that describe visual features that the user desires to have represented in the synthesized image ”) to a second latent representation compatible with the latent representation (Liu, ¶ [0034-0035]: “the trained machine learning diffusion model may be configured to receive user input text … The user input text and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”, see FIG. 2. Liu teaches, using the text encoder – a sub component of the diffusion model, to transform a user input text into feature vector based on the set of embeddings) of the identity (Liu, ¶ [0033]: “The text encoder is configured to receive the set of embeddings (and/or the user identifier) and generate an input feature vector based at least on the set of embeddings”, see FIG. 2); transforming a pose template (Liu, ¶ [0034]: “the trained machine learning diffusion model may include a library of predetermined templates. Each template may include a different combination of embeddings corresponding to a different combination of style and/or scene words that are used to generate the synthesized image”; ¶ [0021]: “the synthesized image includes additional stylized visual features. For example, the character may have different clothes, assume different body poses”; see figures 5 and 6; Liu teaches a library of predetermined templates, each has different style (clothes, body pose, …)) to a set of latent features for the diffusion model (Liu, ¶ [0034]: “The selected template and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”); generating an intermediate image based on the latent representation of the identity, the second latent representation, and the set of latent features (Liu, ¶ [0032-0036]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image”, see FIG. 2. Liu teaches generating a synthesized image from a feature vector which based on (1) embeddings associated with a user identifier, (2) user input text and (3) predefined templated);
Liu does not explicitly disclose modifying, using a face enhancement network, the intermediate image based on the input image;
Zhi-Song Liu is in the same field of art of modifying facial images. Further, Liu-Zhi-Song teaches modifying, using a face enhancement network, the intermediate image based on the input image. (Zhi-Song Liu, page 129117-129118, section C, third paragraph: “For variational autoencoder, it contains an encoder, a decoder and a VGG feature extractor. The encoder learns the latent variable model of the correlation between reference and LR images. Then the decoder samples from the encoded latent space to super-resolve LR images”; See modified figure 5 below. Zhi-Song Liu teaches using a variational autoencoder to enhance an input low-resolution (LR) image, based on a Reference image, to generate a super-resolution (SR) image.)
PNG
media_image5.png
329
1213
media_image5.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu by incorporating the face super-resolution network that is taught by Liu-Zhi-Song, to make a face enhancement network; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve quality of a face image while maintaining original facial features (Liu-Zhi-Song, Abstract: “face super-resolution has still much room to explore good visual quality while preserving original facial attributes … We focus on transferring the key information extracted from reference facial images to the super-resolution process to guarantee the content similarity between the reference and super-resolution image.”).
The combination of Liu and Zhi-Song Liu does not explicitly disclose generating, using a face restoration network, a final output image based on the modified intermediate image;
Ghosh is in the same field of art of modifying face images using diffusion model. Further, Ghosh teaches generating, using a face restoration network, a final output image based on the modified intermediate image. (Ghosh, ¶ [0133-0136 and 0269]: “At operation 410, the interaction system filters artifacts that do not match the user's identity. The interaction system applies a third machine learning model trained to identify and remove artifacts… The facial recognition models extract key features (such as facial features further described herein) from both images and compares them. If significant differences are identified (such as features present in the avatar that do not match the user's image), these facial features are flagged as potential artifacts.”, see FIG. 4. Ghosh teaches a third machine learning (ML) model that receives a modified facial image (by diffusion model) and remove artifacts that are generated from previous image modification processes, thus generate an artifact-free image.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu and Zhi-Song Liu by incorporating the artifact removal ML model that is taught by Ghosh, to make an image modification system that can remove artifacts; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to remove artifact that does not match original identity when modify a face image (Ghosh, ¶ [0133]: “the interaction system filters artifacts that do not match the user's identity”).
The combination of Liu, Zhi-Song Liu and Ghosh then teaches providing for display the final output image on a display of a client device. (Liu, ¶ [0049]: “the social media application displays the synthesized image in the GUI”, see FIG. 4D) (Ghosh, ¶ [0084]: “modify an image as it is captured using device sensors (e.g., one or multiple cameras) of the user system and then displayed on a screen of the user system with the modifications”)
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 2
In regards to Claim 2, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the selfie comprises a self photograph including a representation of a face. (Liu, ¶ [0074]: “capture an image of a face of a user via the camera using the social media application”, see FIG 4A-4B) (Ghosh, ¶ [0099]: “accessing a media content item of a user that includes a face of the user”)
CLAIM 3
In regards to Claim 3, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the text condition comprises textual information that defines a style and content of the intermediate image. (Liu, ¶ [0035]: “The user input text may include style and/or scene words that describe visual features that the user desires to have represented in the synthesized image”) (Ghosh, ¶ [0152]: “A prompt guides the machine learning model's output by setting a specific direction or goal. For instance, if the user's prompt is to generate an avatar in a “cartoon” style, the model will use this information to influence the style of the generated avatar.”)
CLAIM 5
In regards to Claim 5, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the diffusion model comprises a neural network that generates an image from noise based on the text condition. (Liu, ¶ [0036 and 0041]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image of the user based at least on the input feature vector … The Stable Diffusion model processes the input feature vector in an iterative fashion starting with a random starting image information array—e.g., a latent array. The Stable Diffusion model iteratively denoises random noise from the latent array until a designated number of iterations have been reached”) (Ghosh, ¶ [0275]: “the second machine learning model is trained to apply noise to input media content items and then to remove the noise inputted into the media content items to generate modified media content items.”)
CLAIM 6
In regards to Claim 6, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the intermediate image is generated by the diffusion model. (Liu, ¶ [0036 and 0041]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image of the user based at least on the input feature vector …”) (Ghosh, ¶ [0275]: “the second machine learning model is trained to apply noise to input media content items and then to remove the noise inputted into the media content items to generate modified media content items.”)
CLAIM 7
In regards to Claim 7, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the diffusion model, during a denoising process, transforms noise to a particular image based on a text description from the text condition. (Liu, ¶ [0036 and 0041]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image of the user based at least on the input feature vector … The Stable Diffusion model processes the input feature vector in an iterative fashion starting with a random starting image information array—e.g., a latent array. The Stable Diffusion model iteratively denoises random noise from the latent array until a designated number of iterations have been reached”) (Ghosh, ¶ [0275]: “the second machine learning model is trained to apply noise to input media content items and then to remove the noise inputted into the media content items to generate modified media content items.”)
CLAIM 8
In regards to Claim 8, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the second latent representation that is compatible with the diffusion model preserves a set facial attributes of the input image. (Liu, ¶ [0034-0035]: “The user input text and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”, see FIG. 2. Liu teaches transforming a user input text into feature vector based on the set of embeddings which associate with facial features of input image)
CLAIM 10
In regards to Claim 10, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches a face enhancement network improves the identity on the intermediate image using the input image as a reference (Zhi-Song Liu, Zhi-Song Liu, page 129117-129118, section C, third paragraph: “The encoder learns the latent variable model of the correlation between reference and LR images. Then the decoder samples from the encoded latent space to super-resolve LR images”; page 129113, right col, first paragraph: “we can obtain SR image with both sharp visual quality and remain the ground truth identity”; See modified figure 5 in the rejection of claim 1. Zhi-Song Liu teaches using a variational autoencoder to enhance an input low-resolution (LR) image, based on a Reference image, to generate a super-resolution (SR) image. The SR image has better visual quality while preserving identity of the face image), and a face restoration network improves a set of details and reduces a number of artifacts on the intermediate image after being processed by the face enhancement network. (Ghosh, ¶ [0133-0136 and 0269]: “At operation 410, the interaction system filters artifacts that do not match the user's identity. The interaction system applies a third machine learning model trained to identify and remove artifacts… The facial recognition models extract key features (such as facial features further described herein) from both images and compares them. If significant differences are identified (such as features present in the avatar that do not match the user's image), these facial features are flagged as potential artifacts.”, see FIG. 4. Ghosh teaches maintaining facial features that are similar with original face and filtering out artifacts that are different with original face)
CLAIM 11
In regards to Claim 11, Liu teaches a system (Liu, ¶ [0018]: “a computing device for generation of synthetic images using a trained machine learning diffusion model”) comprising: a processor; and a memory (Liu, ¶ [0018]: “The computing device includes a processor (e.g., central processing units, or “CPUs”), volatile memory, non-volatile memory,… ”) including instructions that, when executed by the processor, cause the processor to perform operations (Liu, ¶ [0018]: “The non-volatile memory stores instructions”) comprising: receiving an input image, the input image comprising a selfie (Liu, ¶ [0019-0020]: “capturing an image of a user (referred to as input image … position at least some part of the body of the user (e.g., the face) at a designated location in a field of view of the camera)”; see input image 500 and 600 in figures 5 and 6, respectively); transforming, using a neural network (Liu, ¶ [0028-0029]: “the image encoder includes a vision transformer (ViT)”), the input image to a latent representation (Liu, ¶ [0028-0029]: “The image encoder is configured to receive the image of the user and generate a set of embeddings that semantically describes visual features of the user based at least on the image of the user. The set of embeddings is selected from a lexicon of embeddings within a latent space…” Liu teaches using a vision transformer to transform input image into a set of embeddings) of an identity (Liu, ¶ [0032]: “the set of embeddings generated for the image of the user is associated with a user identifier”, ¶ [0053]: “different user identifiers… determined from the different input images”. Liu teaches generating transforming input images to embeddings that describe facial features of user, associated with an identifier, different input will generate different identifier); transforming, using a diffusion model (Liu, ¶ [0035]: “the trained machine learning diffusion model ”), a text condition (Liu, ¶ [0035]: “The user input text may include style and/or scene words that describe visual features that the user desires to have represented in the synthesized image ”) to a second latent representation compatible with the latent representation (Liu, ¶ [0034-0035]: “the trained machine learning diffusion model may be configured to receive user input text … The user input text and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”, see FIG. 2. Liu teaches, using the text encoder – a sub component of the diffusion model, to transform a user input text into feature vector based on the set of embeddings) of the identity (Liu, ¶ [0033]: “The text encoder is configured to receive the set of embeddings (and/or the user identifier) and generate an input feature vector based at least on the set of embeddings”, see FIG. 2); transforming a pose template (Liu, ¶ [0034]: “the trained machine learning diffusion model may include a library of predetermined templates. Each template may include a different combination of embeddings corresponding to a different combination of style and/or scene words that are used to generate the synthesized image”; ¶ [0021]: “the synthesized image includes additional stylized visual features. For example, the character may have different clothes, assume different body poses”; see figures 5 and 6; Liu teaches a library of predetermined templates, each has different style (clothes, body pose, …)) to a set of latent features for the diffusion model (Liu, ¶ [0034]: “The selected template and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”); generating an intermediate image based on the latent representation of the identity, the second latent representation, and the set of latent features (Liu, ¶ [0032-0036]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image”, see FIG. 2. Liu teaches generating a synthesized image from a feature vector which based on (1) embeddings associated with a user identifier, (2) user input text and (3) predefined templated);
Liu does not explicitly disclose modifying, using a face enhancement network, the intermediate image based on the input image;
Zhi-Song Liu is in the same field of art of modifying facial images. Further, Liu-Zhi-Song teaches modifying, using a face enhancement network, the intermediate image based on the input image. (Zhi-Song Liu, page 129117-129118, section C, third paragraph: “For variational autoencoder, it contains an encoder, a decoder and a VGG feature extractor. The encoder learns the latent variable model of the correlation between reference and LR images. Then the decoder samples from the encoded latent space to super-resolve LR images”; See modified figure 5 below. Zhi-Song Liu teaches using a variational autoencoder to enhance an input low-resolution (LR) image, based on a Reference image, to generate a super-resolution (SR) image.)
PNG
media_image5.png
329
1213
media_image5.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu by incorporating the face super-resolution network that is taught by Liu-Zhi-Song, to make a face enhancement network; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve quality of a face image while maintaining original facial features (Liu-Zhi-Song, Abstract: “face super-resolution has still much room to explore good visual quality while preserving original facial attributes … We focus on transferring the key information extracted from reference facial images to the super-resolution process to guarantee the content similarity between the reference and super-resolution image.”).
The combination of Liu and Zhi-Song Liu does not explicitly disclose generating, using a face restoration network, a final output image based on the modified intermediate image;
Ghosh is in the same field of art of modifying face images using diffusion model. Further, Ghosh teaches generating, using a face restoration network, a final output image based on the modified intermediate image. (Ghosh, ¶ [0133-0136 and 0269]: “At operation 410, the interaction system filters artifacts that do not match the user's identity. The interaction system applies a third machine learning model trained to identify and remove artifacts… The facial recognition models extract key features (such as facial features further described herein) from both images and compares them. If significant differences are identified (such as features present in the avatar that do not match the user's image), these facial features are flagged as potential artifacts.”, see FIG. 4. Ghosh teaches a third machine learning (ML) model that receives a modified facial image (by diffusion model) and remove artifacts that are generated from previous image modification processes, thus generate an artifact-free image.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu and Zhi-Song Liu by incorporating the artifact removal ML model that is taught by Ghosh, to make an image modification system that can remove artifacts; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to remove artifact that does not match original identity when modify a face image (Ghosh, ¶ [0133]: “the interaction system filters artifacts that do not match the user's identity”).
The combination of Liu, Zhi-Song Liu and Ghosh then teaches providing for display the final output image on a display of a client device. (Liu, ¶ [0049]: “the social media application displays the synthesized image in the GUI”, see FIG. 4D) (Ghosh, ¶ [0084]: “modify an image as it is captured using device sensors (e.g., one or multiple cameras) of the user system and then displayed on a screen of the user system with the modifications”)
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 12
In regards to Claim 12, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the selfie comprises a self photograph including a representation of a face. (Liu, ¶ [0074]: “capture an image of a face of a user via the camera using the social media application”, see FIG 4A-4B) (Ghosh, ¶ [0099]: “accessing a media content item of a user that includes a face of the user”)
CLAIM 13
In regards to Claim 13, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the text condition comprises textual information that defines a style and content of the intermediate image. (Liu, ¶ [0035]: “The user input text may include style and/or scene words that describe visual features that the user desires to have represented in the synthesized image”) (Ghosh, ¶ [0152]: “A prompt guides the machine learning model's output by setting a specific direction or goal. For instance, if the user's prompt is to generate an avatar in a “cartoon” style, the model will use this information to influence the style of the generated avatar.”)
CLAIM 15
In regards to Claim 15, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the diffusion model comprises a neural network that generates an image from noise based on the text condition. (Liu, ¶ [0036 and 0041]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image of the user based at least on the input feature vector … The Stable Diffusion model processes the input feature vector in an iterative fashion starting with a random starting image information array—e.g., a latent array. The Stable Diffusion model iteratively denoises random noise from the latent array until a designated number of iterations have been reached”) (Ghosh, ¶ [0275]: “the second machine learning model is trained to apply noise to input media content items and then to remove the noise inputted into the media content items to generate modified media content items.”)
CLAIM 16
In regards to Claim 16, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the intermediate image is generated by the diffusion model. (Liu, ¶ [0036 and 0041]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image of the user based at least on the input feature vector …”) (Ghosh, ¶ [0275]: “the second machine learning model is trained to apply noise to input media content items and then to remove the noise inputted into the media content items to generate modified media content items.”)
CLAIM 17
In regards to Claim 17, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the diffusion model, during a denoising process, transforms noise to a particular image based on a text description from the text condition. (Liu, ¶ [0036 and 0041]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image of the user based at least on the input feature vector … The Stable Diffusion model processes the input feature vector in an iterative fashion starting with a random starting image information array—e.g., a latent array. The Stable Diffusion model iteratively denoises random noise from the latent array until a designated number of iterations have been reached”) (Ghosh, ¶ [0275]: “the second machine learning model is trained to apply noise to input media content items and then to remove the noise inputted into the media content items to generate modified media content items.”)
CLAIM 18
In regards to Claim 18, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches the second latent representation that is compatible with the diffusion model preserves a set facial attributes of the input image. (Liu, ¶ [0034-0035]: “The user input text and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”, see FIG. 2. Liu teaches transforming a user input text into feature vector based on the set of embeddings which associate with facial features of input image)
CLAIM 20
In regards to Claim 20, Liu teaches a non-transitory computer-readable medium comprising instructions (Liu, ¶ [0018]: “The non-volatile memory stores instructions”), which when executed by a computing device (Liu, ¶ [0018]: “a computing device for generation of synthetic images using a trained machine learning diffusion model”), cause the computing device to perform operations comprising: receiving an input image, the input image comprising a selfie (Liu, ¶ [0019-0020]: “capturing an image of a user (referred to as input image … position at least some part of the body of the user (e.g., the face) at a designated location in a field of view of the camera)”; see input image 500 and 600 in figures 5 and 6, respectively); transforming, using a neural network (Liu, ¶ [0028-0029]: “the image encoder includes a vision transformer (ViT)”), the input image to a latent representation (Liu, ¶ [0028-0029]: “The image encoder is configured to receive the image of the user and generate a set of embeddings that semantically describes visual features of the user based at least on the image of the user. The set of embeddings is selected from a lexicon of embeddings within a latent space…” Liu teaches using a vision transformer to transform input image into a set of embeddings) of an identity (Liu, ¶ [0032]: “the set of embeddings generated for the image of the user is associated with a user identifier”, ¶ [0053]: “different user identifiers… determined from the different input images”. Liu teaches generating transforming input images to embeddings that describe facial features of user, associated with an identifier, different input will generate different identifier); transforming, using a diffusion model (Liu, ¶ [0035]: “the trained machine learning diffusion model ”), a text condition (Liu, ¶ [0035]: “The user input text may include style and/or scene words that describe visual features that the user desires to have represented in the synthesized image ”) to a second latent representation compatible with the latent representation (Liu, ¶ [0034-0035]: “the trained machine learning diffusion model may be configured to receive user input text … The user input text and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”, see FIG. 2. Liu teaches, using the text encoder – a sub component of the diffusion model, to transform a user input text into feature vector based on the set of embeddings) of the identity (Liu, ¶ [0033]: “The text encoder is configured to receive the set of embeddings (and/or the user identifier) and generate an input feature vector based at least on the set of embeddings”, see FIG. 2); transforming a pose template (Liu, ¶ [0034]: “the trained machine learning diffusion model may include a library of predetermined templates. Each template may include a different combination of embeddings corresponding to a different combination of style and/or scene words that are used to generate the synthesized image”; ¶ [0021]: “the synthesized image includes additional stylized visual features. For example, the character may have different clothes, assume different body poses”; see figures 5 and 6; Liu teaches a library of predetermined templates, each has different style (clothes, body pose, …)) to a set of latent features for the diffusion model (Liu, ¶ [0034]: “The selected template and the set of embeddings may be provided as input to the text encoder and the text encoder may generate the input feature vector”); generating an intermediate image based on the latent representation of the identity, the second latent representation, and the set of latent features (Liu, ¶ [0032-0036]: “The diffusion model is configured to receive the input feature vector and generate the synthesized image”, see FIG. 2. Liu teaches generating a synthesized image from a feature vector which based on (1) embeddings associated with a user identifier, (2) user input text and (3) predefined templated);
Liu does not explicitly disclose modifying, using a face enhancement network, the intermediate image based on the input image;
Zhi-Song Liu is in the same field of art of modifying facial images. Further, Liu-Zhi-Song teaches modifying, using a face enhancement network, the intermediate image based on the input image. (Zhi-Song Liu, page 129117-129118, section C, third paragraph: “For variational autoencoder, it contains an encoder, a decoder and a VGG feature extractor. The encoder learns the latent variable model of the correlation between reference and LR images. Then the decoder samples from the encoded latent space to super-resolve LR images”; See modified figure 5 below. Zhi-Song Liu teaches using a variational autoencoder to enhance an input low-resolution (LR) image, based on a Reference image, to generate a super-resolution (SR) image.)
PNG
media_image5.png
329
1213
media_image5.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu by incorporating the face super-resolution network that is taught by Liu-Zhi-Song, to make a face enhancement network; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve quality of a face image while maintaining original facial features (Liu-Zhi-Song, Abstract: “face super-resolution has still much room to explore good visual quality while preserving original facial attributes … We focus on transferring the key information extracted from reference facial images to the super-resolution process to guarantee the content similarity between the reference and super-resolution image.”).
The combination of Liu and Zhi-Song Liu does not explicitly disclose generating, using a face restoration network, a final output image based on the modified intermediate image;
Ghosh is in the same field of art of modifying face images using diffusion model. Further, Ghosh teaches generating, using a face restoration network, a final output image based on the modified intermediate image. (Ghosh, ¶ [0133-0136 and 0269]: “At operation 410, the interaction system filters artifacts that do not match the user's identity. The interaction system applies a third machine learning model trained to identify and remove artifacts… The facial recognition models extract key features (such as facial features further described herein) from both images and compares them. If significant differences are identified (such as features present in the avatar that do not match the user's image), these facial features are flagged as potential artifacts.”, see FIG. 4. Ghosh teaches a third machine learning (ML) model that receives a modified facial image (by diffusion model) and remove artifacts that are generated from previous image modification processes, thus generate an artifact-free image.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu and Zhi-Song Liu by incorporating the artifact removal ML model that is taught by Ghosh, to make an image modification system that can remove artifacts; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to remove artifact that does not match original identity when modify a face image (Ghosh, ¶ [0133]: “the interaction system filters artifacts that do not match the user's identity”).
The combination of Liu, Zhi-Song Liu and Ghosh then teaches providing for display the final output image on a display of a client device. (Liu, ¶ [0049]: “the social media application displays the synthesized image in the GUI”, see FIG. 4D) (Ghosh, ¶ [0084]: “modify an image as it is captured using device sensors (e.g., one or multiple cameras) of the user system and then displayed on a screen of the user system with the modifications”)
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 4, 9, 14 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Zhi-Song Liu in view of Ghosh, and further in view of Colado et al. (Colado, Iván J. Pérez, et al. "Using new AI-driven techniques to ease serious games authoring." 2023 IEEE Frontiers in Education Conference (FIE), hereinafter Colado).
CLAIM 4
In regards to Claim 4, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1.
The combination of Liu, Zhi-Song Liu and Ghosh does not explicitly disclose the pose template comprises an image that includes a geometry that is to be applied for generating the intermediate image.
PNG
media_image6.png
250
1216
media_image6.png
Greyscale
Colado is in the same field of art of diffusion model. Further, Colado teaches the pose template comprises an image that includes a geometry that is to be applied for generating the intermediate image. (Colado, page 5, subsection a) Creation of a Character Concept Sheet, see reconstructed text and modified figure 2 below. Colado teaches different pose templates comprise images of human key points on black background from OpenPose, and using ControlNet to apply the poses on generated characters, see Fig. 2-4)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu, Zhi-Song Liu and Ghosh by incorporating the ControlNet + OpenPose framework that is taught by Colado, to make an diffusion-based image generation system that can control the pose of generated human image; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to simplifying the process of controlling poses of generated human image (Colado, page 5, section 2 Character Generation: “Generating characters for a serious game presents various challenges, such as controlling the character's pose”; Abstract: “this paper presents a working methodology to simplify the development of serious games”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 9
In regards to Claim 9, the combination of Liu, Zhi-Song Liu and Ghosh teaches the method of Claim 1. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches transform image information into a set of latent representations that are used by the diffusion model. (Liu, ¶ [0015]: “the image encoder is pre-trained to project general image information into a latent space rich in semantic information that is used to control the trained diffusion model”)
The combination of Liu, Zhi-Song Liu and Ghosh does not explicitly disclose geometry information from the pose template is utilized to transform a geometry into a set of latent representations that are used by the diffusion model such that the final output image has substantially similar geometry to the pose template.
PNG
media_image7.png
1245
1183
media_image7.png
Greyscale
Colado is in the same field of art of diffusion model. Further, Colado teaches geometry information from the pose template is utilized to transform a geometry into a set of latent representations that are used by the diffusion model such that the final output image has substantially similar geometry to the pose template. (Colado, page 5, subsection a) Creation of a Character Concept Sheet, see figure 2 below. Colado teaches using Stable Diffusion with ControlNet to transform pose templates to latent representations, and apply the poses on generated characters.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu, Zhi-Song Liu and Ghosh by incorporating the ControlNet + OpenPose framework that is taught by Colado, to make an diffusion-based image generation system that can control the pose of generated human image; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to simplifying the process of controlling poses of generated human image (Colado, page 5, section 2 Character Generation: “Generating characters for a serious game presents various challenges, such as controlling the character's pose”; Abstract: “this paper presents a working methodology to simplify the development of serious games”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 14
In regards to Claim 14, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11.
The combination of Liu, Zhi-Song Liu and Ghosh does not explicitly disclose the pose template comprises an image that includes a geometry that is to be applied for generating the intermediate image.
PNG
media_image6.png
250
1216
media_image6.png
Greyscale
Colado is in the same field of art of diffusion model. Further, Colado teaches the pose template comprises an image that includes a geometry that is to be applied for generating the intermediate image. (Colado, page 5, subsection a) Creation of a Character Concept Sheet, see reconstructed text and modified figure 2 below. Colado teaches different pose templates comprise images of human key points on black background from OpenPose, and using ControlNet to apply the poses on generated characters, see Fig. 2-4)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu, Zhi-Song Liu and Ghosh by incorporating the ControlNet + OpenPose framework that is taught by Colado, to make an diffusion-based image generation system that can control the pose of generated human image; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to simplifying the process of controlling poses of generated human image (Colado, page 5, section 2 Character Generation: “Generating characters for a serious game presents various challenges, such as controlling the character's pose”; Abstract: “this paper presents a working methodology to simplify the development of serious games”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 19
In regards to Claim 19, the combination of Liu, Zhi-Song Liu and Ghosh teaches the system of Claim 11. In addition, the combination of Liu, Zhi-Song Liu and Ghosh teaches transform image information into a set of latent representations that are used by the diffusion model. (Liu, ¶ [0015]: “the image encoder is pre-trained to project general image information into a latent space rich in semantic information that is used to control the trained diffusion model”)
The combination of Liu, Zhi-Song Liu and Ghosh does not explicitly disclose geometry information from the pose template is utilized to transform a geometry into a set of latent representations that are used by the diffusion model such that the final output image has substantially similar geometry to the pose template.
Colado is in the same field of art of diffusion model. Further, Colado teaches geometry information from the pose template is utilized to transform a geometry into a set of latent representations that are used by the diffusion model such that the final output image has substantially similar geometry to the pose template. (Colado, page 5, subsection a) Creation of a Character Concept Sheet, see figure 2 below. Colado teaches using Stable Diffusion with ControlNet to transform pose templates to latent representations, and apply the poses on generated characters.)
PNG
media_image7.png
1245
1183
media_image7.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Liu, Zhi-Song Liu and Ghosh by incorporating the ControlNet + OpenPose framework that is taught by Colado, to make an diffusion-based image generation system that can control the pose of generated human image; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to simplifying the process of controlling poses of generated human image (Colado, page 5, section 2 Character Generation: “Generating characters for a serious game presents various challenges, such as controlling the character's pose”; Abstract: “this paper presents a working methodology to simplify the development of serious games”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Pertinent Arts
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
ThinkDiffusion (“ControlNet OpenPose” https://learn.thinkdiffusion.com/controlnet-openpose/, a 2023 archived copy is attached, hereinafter ThinkDiffusion) which is directed to a method using ControlNet and OpenPose to control and manipulate human poses within the Stable Diffusion framework.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHUT HUY (JEREMY) PHAM whose telephone number is (703)756-5797. The examiner can normally be reached Mo - Fr. 8:30am - 6pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O'Neal Mistry can be reached on (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHUT HUY PHAM/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights