DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6-8, and 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (Chen, Jingye, et al. "Textdiffuser: Diffusion models as text painters." Advances in Neural Information Processing Systems 36 (2023): 9353-9387, hereinafter “Chen”) in view of Iyer et al. (US 20240338870 A1, hereinafter “Iyer”) and Xiao et al. (Xiao, Jie, et al. "Ccm: Adding conditional controls to text-to-image consistency models." arXiv preprint arXiv:2312.06971 (2023)., hereinafter “Xiao”).
Regarding claim 6,
Chen teaches:
A system comprising: at least one memory storing programmed instructions;
one or more Input /Output (1/O) interfaces; and one or more hardware processors operatively coupled to the at least one memory (Chen p5 section 5.1, “. . . implement the diffusion process . . . We set the batch size to 768 and trained the model for two epochs, taking four days using 8 Tesla V100 GPUs with 32GB memory. . .”; Abstract, “. . . controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. . .”; NOTE: The system is the computing system that houses the Tesla GPUs coupled with memory to implement a diffusion process. The input interface is inherent to the system because the system needs a text prompt, such as an interface where a user can type. The output interface is inherent in order to display the generated images, such as the Textdiffuser output images illustrated in Fig. 4),
wherein the one or more hardware processors are configured by the programmed instructions to:
receive a data comprising an input image, a textual prompt, a mask representing a Region of Interest (Rol) in the input image (Chen: Abstract, “. . . create high-quality text images using text prompts . . .”; p3 section 3.1, “. . . we utilize the Transformer architecture to obtain the layout of keywords. Formally, we denote the tokenized prompt as 𝒫=(p0,p1,…,pL−1), where L means the maximum length of tokens . . .”; p4 section 3.2, “. . . In the process of part-image generation (also called text inpainting), the feature mask M^ represents the region where the user wants to generate, while the masked feature F^M indicates the region that the user wants to preserve. . .”; NOTE: The process of inpainting requires an image input where the masks are located and represented as feature mask M^ and masked feature F^M.),
generate a character mask pertaining to the input image based on the plurality of control parameters using a character mask generation technique (Chen: p3 section 3.1, “. . . utilize some Python packages like Pillow to render the texts and meanwhile obtain the character-level segmentation mask 𝐂 with |𝒜| channels, where |𝒜| denote the size of alphabet 𝒜. To this end, we obtain the layouts of keywords. . .”; NOTE: The A channels are the parameters that are used to obtain the segmentation mask C, constituting to the character mask pertaining to the input image.);
generate a generation mask comprising a plurality of character regions and a plurality of non-character regions based on the character mask, wherein the plurality of character regions are marked as one and the plurality of non-character regions are marked as zero (Chen: p25 Appendix J, “. . . set the value of character pixels to 1 and non-character pixels to 0 (i.e., remove the content and only provide the position guidance) . . .”; NOTE: As discussed above, segmentation mask C are the character regions, and feature mask M^ and masked feature F^M are non-character masks corresponding to regions where to generate, and what regions to preserve. Appendix J teaches that pixels corresponding to characters, such as characters in the segmentation mask C are marked as 1, and non-character pixels in regions outside the character mask are marked as 0.),
wherein a bounding box is generated on each of the plurality of character regions using a renderer (Chen: pg3 section 3.1, “. . . utilize bounding boxes to determine the layout of keywords (enclosed with quotes specified by user prompts). . . utilize some Python packages like Pillow to render the texts);
generate a conditional mask pertaining to the input image based on the generated bounding box and the plurality of control parameters using the renderer (Chen: p2 section 1, “. . . fine-tune the latent diffusion model by leveraging the generated segmentation masks as conditions for the diffusion process . . .”; p4 section 3.2, “. . . to generate the image guided by the segmentation masks . . . use VAE [35] to encode the original image with shape H×W into 4-D latent space features 𝐅∈ℝ4×H′×W′ . . .”; NOTE: The parameters are H and W);
generate a customized textual image based on the character mask and the conditional mask associated with the input image using a textual image generating diffusion model by (Chen: Abstract, “. . . TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout . . .”; NOTE: Figure 2 illustrates Chen’s pipeline from a user prompt that is used to generate the customized textual image outputted by the diffusion model that uses input images the text prompt including the masks discussed above. Because the diffusion model generates a textual image, for example, the phrase “hello world” held by a cat, therefore, Chen’s diffusion model is a textual image generating diffusion model.):
initializing the textual image generating diffusion model with a random gaussian noise, the character mask, and the generation mask (Chen: p4 section 3.2, “generate the image guided by the segmentation masks 𝐂 . . . sample a Gaussian noise ϵ∈ℝ4×H′×W′ to corrupt the original feature, yielding F^ . . . We concatenate F^,C^,M^,F^M in the feature channel as a 17-D input and use denoising loss between the sampled noise ϵ and the predicted noise ϵθ,; NOTE: Sampling from a Gaussian distribution is a process of sampling a random Gaussian noise, the result from the Gaussian noise sampling F^ is concatenated with the character mask, and the generation mask, “F^,C^,M^,F^M” and are initialized.);
generating an intermediate image based on the input image and the random gaussian noise (Chen: p4 section 3.2, “. . . sample a time step T∼Uniform(0,Tmax) and sample a Gaussian noise . . . to corrupt the original feature, yielding F . . .”; NOTE: The image with corrupted original feature via a Gaussian noise is the intermediate image.);
iteratively refining the intermediate image for a predefined number of timesteps to generate a latent vector image (Chen: p4 section 3.2, “. . . sample a time step T∼Uniform(0,Tmax). . . “; p6 section 5.1, “. . . During training, we follow [25] to set the maximum time step Tmax to 1,000, and the caption is dropped with a probability of 10% for classifier-free guidance [27]. When training the part-image generation branch, the detected text box is masked with a likelihood of 50%. We use 50 sampling steps during inference and classifier-free guidance with a scale of 7.5 following . . .”; NOTE: The iteration is an iterative loop of the sampling predefined at 50 sampling steps, the latent vector image is the image generated after the 50 sampling steps are done); and
generating the customized textual image from the latent vector image (Chen: p4 section 3.2, “. . . the output features are fed into the VAE decoder to obtain the images)
Although Chen teaches: generate a conditional mask pertaining to the input image based on the generated bounding box and the plurality of control parameters using the renderer, and outputting customized textual images with different font styles, font colors, and background as shown in Fig. 4, Chen fails to disclose: a plurality of control parameters for governing manipulation of font colour font type, and background of the input image; and generate a conditional mask pertaining to the input image based on the plurality of control parameters.
The analogous art Iyer teaches:
a plurality of control parameters for governing manipulation of font colour, font type, and background of the input image (Iyer: ¶59, “At operation 210, the system identifies one or more styling parameters. In some cases, the operations of this step refer to, or may be performed by, an image processing apparatus as described with reference to FIGS. 1 and 8. In some cases, the one or more styling parameters include a text effect . . . font . . . text color, etc. . .”; ¶117, “. . .image generation model 855 identifies a background color, where the output image is generated based on the background color. . .”);
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Chen, and Iyer, and include: a plurality of control parameters for governing manipulation of font colour, font type, and background of the input image.
The reason for doing so to “address the challenge of efficiently applying intricate, detailed, and visually interesting text effects, referred to as “Text Effects,” in a design workflow. Traditional approaches to creating text effects can be time-consuming, and generating effects that are similar in style but not identical while maintaining detail fidelity and visual interest can be particularly difficult”, to “quickly generate multiple detailed and consistent text effects, which can be used as is or as a starting point for further refinement”, and to “provide a fast and reliable approach for stylizing text while maintaining legibility and consistent styling. The generation creativity, text legibility, and stylization consistency are all highly controllable” (Iyer: ¶31).
Although Iyer discloses maintaining style consistency by using the same style text embedding to guide the generation of each separate glyph, and that an image generation network can be trained on both image embeddings and text captions, and although Chen teaches character segmentation masks defining the regions of the characters, the combination of Chen, and Iyer fails to disclose: generating the customized textual image from the latent vector image using a trained customized character map-guided consistency model. (NOTE: As discussed above, Chen uses a VAE decoder to generate the latent vector image. Chen also addresses the issues with VAE decoder when reconstructing images with small characters (p10 Section 6) resulting in losing details (p 34. Appendix Q),
The analogous art Xiao teaches:
A consistency model that can generate high-quality images efficiently based on a trained ControlNet branch conditioned on conditional controls include edge, depth, and masked image, (NOTE: which defines spatial regions) (Xiao: Abstract: “Consistency Models (CMs) have showed a promise in creating visual content efficiently and with high quality. . . technical report, we consider alternative strategies for adding ControlNet-like conditional control to CMs and present three significant findings. 1) ControlNet trained for diffusion models (DMs) can be directly applied to CMs for high-level semantic controls but struggles with low-level detail and realism control. 2) CMs serve as an independent class of generative models, based on which ControlNet can be trained from scratch using Consistency Training proposed by Song et al [25, 23]. 3) A lightweight adapter can be jointly optimized under multiple conditions through Consistency Training, allowing for the swift transfer of DMs-based ControlNet to CMs. We study these three solutions across various conditional controls, including edge, depth, human pose, low-resolution image and masked image”).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Chen, Iyer, and Xiao: and substitute Chen’s VAE decoder to Xiao Consistency model, which are both generative models. Xiao’s CM can be trained via ControlNet using masked images, such as the segmentation character mask of Chen. Using Chen’s segmentation mask C as input to Xiao’s CM results in a predictable character map-guided CM model, because Chen’s segmentation masks are the masks for the character for: generating the customized textual image from the latent vector image (as taught by Chen) using a trained customized character map-guided consistency model (as taught by Xiao)
The reason for doing so is to “generate high-quality images in one or few steps” (Xiao: p1 section 1.)
Regarding claim 7, depending on 6,
The combination of Chen, Iyer, and Xiao, teaches:
The system of claim 6,
Chen further teaches:
wherein generating the character mask pertaining to the input image based on the plurality of control parameters using the character mask generation technique comprises:
extracting at least one text to be written on the Rol of the input image from the textual prompt using lexical filtering; predicting the bounding box of the extracted text using a Layout Transformer (Chen: p3 section 3.1, “. . . the objective is to utilize bounding boxes to determine the layout of keywords (enclosed with quotes specified by user prompts). Inspired by Layout Transformer [20], we utilize the Transformer architecture to obtain the layout of keywords. Formally, we denote the tokenized prompt as 𝒫=(p0,p1,…,pL−1), where L means the maximum length of tokens . . .”; NOTE: The generated tokens based on the prompt are the text to be written on the RoI of the input image, for example shown in figure 4, the prompt includes “hello world”, and the RoI is the board where the tokens hello, world, are to be written. Because the tokens are generated from the textual prompt words, therefore, the process of tokenizing the prompt is a process of lexical filtering.);
and generating the character mask based on the bounding box using a renderer (Chen: p3 section 3.1, “. . . utilize some Python packages like Pillow to render the texts and meanwhile obtain the character-level segmentation mask 𝐂),
wherein character regions are marked with positive values and the non-character regions are marked with zeros (Chen: p25 Appendix J, “. . . set the value of character pixels to 1 and non-character pixels to 0. . .”; NOTE: The pixels defining the character is the character region, and the pixels defining the non-characters is the non-character region. The number “1” is a positive value, a character consisting multiple pixels will have positive values.).
Regarding claim 8, depending on 6,
The combination of Chen, Iyer, and Xiao, teaches:
The system of claim 6,
Chen further teaches:
wherein the textual prompt comprises a plurality of requirements pertaining to customization (NOTE: Figure 4 illustrates multiple example textual prompts. One example is the textual prompt “a bear holds a board saying 'hello world'”, the requirement for the image is to have a bear, a board, and the words saying “hello world” on a board held by a bear. Another example textual prompt “a boy holds 'P' and a girl holds 'Q'”, the requirement is that a boy holds P, and a girl holds Q.)
Regarding method claims 1-3, method claims 1-3 are drawn to the methods corresponding to the programmed instructions of using same as claimed in apparatus claims 6-8. Therefore, method claims 1-3 correspond to the programmed instructions in the apparatus of claims 6-8, and are rejected for the same reasons of obviousness as used above
Regarding CRM claims 11-13, method claims 11-13 are drawn to the CRM corresponding to the programmed instructions of using same as claimed in apparatus claims 6-8. Therefore, CRM claims 11-13 correspond to the programmed instructions in the apparatus of claims 6-8, and are rejected for the same reasons of obviousness as used above.
Allowable Subject Matter
Claims 4-5, 9-10, and 14-15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights