Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restriction
Restriction to one of the following inventions is required under 35 U.S.C. 121:
I. Claims 1-14, drawn to generating a synthesized digital image based on a color conditioning input and image prompt using a diffusion neural network, classified in G06N3/0475.
II. Claims 15-20, drawn to training a diffusion neural network based on generating a super-pixel image and a predicted noise vector, classified in G06N3/08.
The inventions are independent or distinct, each from the other because:
Inventions I and II are related as subcombinations disclosed as usable together in a single combination. The subcombinations are distinct if they do not overlap in scope and are not obvious variants, and if it is shown that at least one subcombination is separately usable. In the instant case, subcombination II has separate utility such as training a diffusion neural network. See MPEP § 806.05(d).
The examiner has required restriction between subcombinations usable together. Where applicant elects a subcombination and claims thereto are subsequently found allowable, any claim(s) depending from or otherwise requiring all the limitations of the allowable subcombination will be examined for patentability in accordance with 37 CFR 1.104. See MPEP § 821.04(a). Applicant is advised that if any claim presented in a divisional application is anticipated by, or includes all the limitations of, a claim that is allowable in the present application, such claim may be subject to provisional statutory and/or nonstatutory double patenting rejections over the claims of the instant application.
Restriction for examination purposes as indicated is proper because all the inventions listed in this action are independent or distinct for the reasons given above and there would be a serious search and/or examination burden if restriction were not required because one or more of the following reasons apply:
The two inventions are drawn to mutually exclusive characteristics, as described above; thus, a search of one invention would be fundamentally different from a search for the other invention. Prior art found for one invention would in no way be necessarily related to the other invention; and thus, two entirely different searches would be required (including searching different classes/subclasses). For example, a search for Invention I, which requires an image prompt and color conditioning input for generating a synthesized digital image, would not necessarily yield prior art relevant to, for example, Invention II, which requires a super-pixel image and predicted noise vector for modifying parameters of a diffusion neural network.
Applicant is advised that the reply to this requirement to be complete must include (i) an election of an invention to be examined even though the requirement may be traversed (37 CFR 1.143) and (ii) identification of the claims encompassing the elected invention.
The election of an invention may be made with or without traverse. To reserve a right to petition, the election must be made with traverse. If the reply does not distinctly and specifically point out supposed errors in the restriction requirement, the election shall be treated as an election without traverse. Traversal must be presented at the time of election in order to be considered timely. Failure to timely traverse the requirement will result in the loss of right to petition under 37 CFR 1.144. If claims are added after the election, applicant must indicate which of these claims are readable upon the elected invention.
Should applicant traverse on the ground that the inventions are not patentably distinct, applicant should submit evidence or identify such evidence now of record showing the inventions to be obvious variants or clearly admit on the record that this is the case. In either instance, if the examiner finds one of the inventions unpatentable over the prior art, the evidence or admission may be used in a rejection under 35 U.S.C. 103 or pre-AIA 35 U.S.C. 103(a) of the other invention.
During a telephone conversation with Attorney Ryan Keller on 08/11/2026, a provisional election was made without traverse to prosecute the invention of Invention I, claims 1-14. Affirmation of this election must be made by applicant in replying to this Office action. Claims 15-20 are withdrawn from further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention.
Specification
The disclosure is objected to because of the following informalities: in para 144, “down-sampled features 1425 features have a resolution” should read “down-sampled features 1425 Appropriate correction is required.
Claim Objections
Claim 3 is objected to because of the following informalities: “at one or pixel coordinates” should read “at one or more pixel coordinates…” Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 5 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5 recites the limitation “the digital image”, which renders the claim indefinite. The metes and bounds of what is considered “the digital image” in claim 5 is not defined by the claim or the specification, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
More specifically, “the digital image” in claim 5 seems to refer back to the “digital image” related to the conditions set out by the text description and color conditioning input in claim 1. Because it contains the conditions set out for the generated image, the “digital image” in claim 1 represents aspects of “the synthesized digital image”. However, in claim 5, the synthesis performed by the diffusion network relies on the edges detected in the digital image, thus making it unclear how the digital image in claim 5 also depicts the text description before the synthesis is performed. In view of the foregoing, the metes and bounds of what is considered “the digital image” in claim 5 is unclear.
For examination purposes, “the digital image” in claim 5 will be interpreted to be a digital image input to the edge detection neural network before the synthesized digital image is generated, since synthesis of the synthesized digital image relies on the detected edges, and is not required to represent the text description from claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Liang et al. (Liang, Z., Li, Z., Zhou, S., Li, C., & Loy, C. C. (16 Feb 2024). Control Color: Multimodal Diffusion-Based Interactive Image Colorization. arXiv:2402.10855v1 [cs.CV].), hereinafter Liang, in view of Muhammad et al. (U.S. Patent No. 2024/0354823 A1), hereinafter Muhammad.
Regarding claim 1, Liang teaches a method comprising:
receiving an image prompt comprising a text description of a digital image and a color conditioning input defining a position of a color value for the digital image (Liang, see text prompt and strokes input in FIG. 3 on pg. 3 and control combination in FIG. 1 on pg. 1, last para on pg. 2: “Our framework supports prompt-based, stroke based, exemplar-based, and a combination of these conditions for local and global colorization”);
generating, utilizing a diffusion neural network to process the image prompt and the color conditioning input (Liang, pg. 4, section 3.1: “LDM [36] is proposed to run the diffusion process in latent space instead of pixel space, which saves computational costs”; see FIG. 3 where diffusion model processes the inputs), a synthesized digital image depicting content corresponding to the text description and comprising pixels reflecting the color value at the position (Liang, see FIGs. 1 and 3 examples attached below – outputted images correspond to the stroke colors at their respective positions and the text prompt content; see prompt and stroke control sections on pg. 4, section 3.2.2).
PNG
media_image1.png
297
237
media_image1.png
Greyscale
From Figure 1
PNG
media_image2.png
369
664
media_image2.png
Greyscale
From Figure 3
Liang fails to explicitly teach wherein the image prompt is received from a client device and providing the synthesized digital image for display on the client device. However, Muhammad teaches a method/system for generating a synthesized image (Muhammad, para 3: “In some examples, the image generation model is a diffusion model. An image request input is received (e.g., text input, drawing input, and/or voice input) and, based on the received image request input, an image is generated using the trained image generation model”), wherein the image prompt is received from a client device (Muhammad, para 29: “For example, the external device 232 presents a representation of an item for sale and receives an input. The external device 232 transmits the received input to the computing device 202 via the network 234.”) and providing the synthesized digital image for display on the client device (Muhammad, para 29: “The computing device 202 generates an image using an AI model, such as the AI model 220, corresponding to the received input in a selected style. The computing device 202 transmits the generated image to the external device 232 via the network 234, which then presents the generated image on an interface”; display further described in para 32, 70, 84-85).
Muhammad describes a system and steps for a user to interact with a computing device in order to carry out the claimed method. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the client device and display system, taught by Muhammad, with the method of Liang in order to aid the user in synthesizing images by providing a means for the user to interact with the diffusion neural network and its outputs (See citations from Muhammad above).
Regarding claim 2 (dependent on claim 1), Liang in view of Muhammad teaches further comprising conditioning the diffusion neural network by processing the color conditioning input using a color control adapter (Liang, encoders that process the colored stroke information, see FIG. 1; pg. 4, section 3.2.2: “The strokes are first directly overlaid on the L channel images, denoted as hint images, so that we can get both the color and the position of the strokes. A binary mask is then derived from the hint image. The input image…and the hint image…are encoded into the latent space to get the input latent feature…and the hint latent feature”).
Regarding claim 3 (dependent on claim 1), Liang in view of Muhammad teaches wherein receiving the color conditioning input comprises receiving, from the client device, a condition image that depicts pixels in the color value at one or more pixel coordinates defining the position (Liang, image with color strokes, see examples in FIGs. 1 and 3 where the color values at certain pixel positions are reflected in those areas in the final outputted image).
Regarding claim 4 (dependent on claim 1), Liang in view of Muhammad teaches wherein receiving the color conditioning input comprises receiving, from the client device, a condition image (Liang, image with colored strokes, see examples in FIGs. 1 and 3) depicting: pixels depicting a gray color value at one or more pixel coordinates defining the position; and pixels depicting non-gray color values at pixel coordinates other than the one or more pixel coordinates defining the position (Liang, The strokes image depicts 1) gray color values (non-hint pixels) at pixel coordinates that surround the positions where color is to be reflected, thus defining those positions and 2) non-gray color values (color hints) at the remaining positions, which are positions other than those surrounding the positions where color is to be reflected).
Regarding claim 6 (dependent on claim 1), Liang in view of Muhammad teaches wherein receiving the color conditioning input comprises receiving a condition image of a design template depicting the color value at the position (Liang, the image with color strokes is a design template reflecting a color design for the synthetic image, see examples in FIGs. 1 and 3 where the color values at certain pixel positions are reflected in those areas in the final outputted image).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Liang in view of Muhammad, in further view of Vemuru (Vemuru, K. V. (2022). Implementation of the canny edge detector using a spiking neural network. Future Internet, 14(12), 371.).
Regarding claim 5 (dependent on claim 1), Liang in view of Muhammad teaches wherein generating the synthesized digital image comprises: detecting edges using edge detection; and conditioning synthesis of the diffusion neural network using the edges together with the color conditioning input (Liang, ControlNet detects edges and spatial information in order to condition the synthesis of the diffusion network based on the input color conditions and image; see ControlNet’s implementation in section 3 on pg. 4).
However, Liang in view of Muhammad fails to explicitly teach detecting edges using an edge detection neural network (emphasis added). However, Vemuru teaches a method for detecting edges using an edge detection neural network (Vemuru, pg. 10, section 6: “In conclusion, we present a spiking neural network (SNN) implementation of the Canny edge detector as its neuromorphic analogue by introducing algorithms for spike based computation in the five steps of the conventional algorithm with the conductance based Hodgkin–Huxley neuron as the building block.”; see an example in FIG. 3 on pg. 7).
Liang in view of Muhammad discloses a method for edge detection (Liang’s method requires a control method that detects the edges of objects in the original input image and colors them according to their boundaries), but does not explicitly disclose an edge detection neural network. Vemuru discloses an edge detection neural network that performs Canny edge detection. Thus, each disclose a method for performing edge detection. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the edge detection method taught by Liang in view of Muhammad could have been substituted for (and performed by) that of Vemuru because both serve the purpose of detecting edge features in an image. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the edge detection method of Liang in view of Muhammad for the edge detection neural network taught by Vemuru according to known methods to yield the predictable result of generating a synthesized image with accurate colorization results using detected edges from a model specifically trained on edge detection. Additionally, it is a known technique to input Canny edge detection results to ControlNet to condition image synthesis.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Liang in view of Muhammad, in further view of Peong et al. (Peong, K., Uchida, S., & Haraguchi, D. (2024, August). Typographic text generation with off-the-shelf diffusion model. In International Conference on Document Analysis and Recognition (pp. 52-69). Cham: Springer Nature Switzerland.), hereinafter Peong, and Vemuru.
Regarding claim 7 (dependent on claim 1), Liang in view of Muhammad fails to explicitly teach wherein generating the synthesized digital image further comprises: receiving, from the client device, the color conditioning input comprising a text design element; generating a padded text effect, wherein generating the padded text effect comprises padding edges depicted in the text design element using an edge detection neural network; and conditioning synthesis of the diffusion neural network using the padded text effect together with the color conditioning input.
However, Peong similarly teaches a controlled diffusion model (Peong, abstract: “This paper proposes a typographic text generation system to add and modify text on typographic designs while specifying font styles, colors, and text effects. The proposed system is a novel combination of two off-the-shelf methods for diffusion models, ControlNet and Blended Latent Diffusion”), wherein generating a synthesized digital image further comprises:
receiving a color conditioning input comprising a text design element (Peong, target text image with black text, see FIG. 2, attached below);
generating a padded text effect, wherein generating the padded text effect comprises padding edges depicted in the text design element using an edge detection model (Peong, see FIG. 2 and section 3.5 on pg. 10); and conditioning synthesis of the diffusion neural network using the padded text effect together with the color conditioning input (Peong, generated text image includes black text with shadows padding the edges of the text, implemented by ControlNet, pg. 6, 1st para: “We utilize this function for adding various visual effects (such as “3-D” and “shadows”) around the generated texts”; see generated text in FIG. 2).
PNG
media_image3.png
287
569
media_image3.png
Greyscale
Liang teaches wherein a user can add colored strokes to control the color synthesized by the diffusion model (see color conditioning input in claim 1 rejection). In the invention described by Liang in view of Muhammad and Peong, adding color strokes to the area where text is in the image results in a color conditioning input comprising a text design element, as recited above.
Liang in view of Muhammad and Peong fails to explicitly teach detecting edges using an edge detection neural network (emphasis added). However, Vemuru teaches a method for detecting edges using an edge detection neural network (Vemuru, pg. 10, section 6: “In conclusion, we present a spiking neural network (SNN) implementation of the Canny edge detector as its neuromorphic analogue by introducing algorithms for spike based computation in the five steps of the conventional algorithm with the conductance based Hodgkin–Huxley neuron as the building block.”; see an example in FIG. 3 on pg. 7).
Liang in view of Muhammad and Peong discloses a method for edge detection (Canny edge detector described in pg. 10, section 3.5 of Peong), but does not disclose an edge detection neural network. Vemuru discloses an edge detection neural network that performs Canny edge detection. Thus, each disclose a method for performing edge detection. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the edge detection model taught by Liang in view of Muhammad and Peong could have been substituted for that of Vemuru because both serve the purpose of detecting edge features in an image. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the edge detection model of Liang in view of Muhammad and Peong for the edge detection neural network taught by Vemuru according to known methods to yield the predictable result of generating a synthesized image with accurate colorization results using detected edges from a model specifically trained on edge detection. Additionally, it is a known technique to input Canny edge detection results to ControlNet to condition image synthesis, as demonstrated by Peong’s method.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (Cheng, S. I., Chen, Y. J., Chiu, W. C., Tseng, H. Y., & Lee, H. Y. (2023, January). Adaptively-realistic image generation from stroke and sketch with diffusion model. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 4043-4051). IEEE.), hereinafter Cheng, in view of Muhammad, in further view of Liang.
Regarding claim 8, Cheng teaches a system comprising: a memory component; and one or more processing devices coupled to the memory component, the one or more processing devices to perform operations (Cheng, computer vision hardware to perform the disclosed image processing and used for training and execution of the diffusion model) comprising:
receiving a color conditioning input defining a position of a color value for a digital image (Cheng, color stroke input shown in FIG. 2, attached below, pg. 4055, last para above section 2: “The proposed framework enables a three-dimensional control over image synthesis with flexibility and controllability over shape, color, and realism of the generation, given the input stroke and sketch”);
modifying a diffusion neural network by injecting the color conditioning input into layers of the diffusion neural network using a color control adapter (Cheng, color conditions are used to train the diffusion model, pg. 4057, section 3.2: “In practice, we adopt a two-stage training strategy. First, we train the model with complete sketches and strokes as conditions.”; steps to concatenate sketch and stroke color information is a color control adapter, pg. 4057, section 3.2: “our proposed method concatenates the sketch condition csketch and the stroke condition cstroke along with xt as input for the U-Net model (which is responsible for posterior prediction).”);
generating, utilizing the diffusion neural network conditioned on the color conditioning input (Cheng, pg. 4056, section 3: “In the following we sequentially de scribe our proposed method, starting from the preliminaries for diffusion models (Section 3.1) and the modifications we make for realizing the conditional generation and the discussion for the sketch and stroke guidance (enabled by the technique of classifier-free diffusion guidance, Section 3.2)”), a synthesized digital image comprising pixels reflecting the color value at the position (Cheng, see FIG. 4 on pg. 4058 wherein the color stroke input is reflected at the corresponding position in the synthetically generated images).
PNG
media_image4.png
272
700
media_image4.png
Greyscale
Cheng fails to explicitly teach the following: 1) where the image prompt is received from a client device and providing the synthesized digital image for display on the client device and 2) where the system also receives an image prompt comprising a text description of the digital image and where the synthesized digital image depicts content corresponding to the text description.
Muhammad teaches a method/system for generating a synthesized image (Muhammad, para 3: “In some examples, the image generation model is a diffusion model. An image request input is received (e.g., text input, drawing input, and/or voice input) and, based on the received image request input, an image is generated using the trained image generation model”), wherein the image prompt is received from a client device (Muhammad, para 29: “For example, the external device 232 presents a representation of an item for sale and receives an input. The external device 232 transmits the received input to the computing device 202 via the network 234.”) and providing the synthesized digital image for display on the client device (Muhammad, para 29: “The computing device 202 generates an image using an AI model, such as the AI model 220, corresponding to the received input in a selected style. The computing device 202 transmits the generated image to the external device 232 via the network 234, which then presents the generated image on an interface”; display further described in para 32, 70, 84-85).
Muhammad describes a system and steps for a user to interact with a computing device in order to carry out the claimed method. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the client device and display, taught by Muhammad, with the system of Cheng in order to aid the user in synthesizing images by providing a means for the user to interact with the diffusion neural network and its outputs (See citations from Muhammad above).
Additionally, Liang similarly teaches a color stroke method for synthesizing images using a diffusion network. Liang teaches the use of ControlNet to control a fixed diffusion model (Liang, see ControlNet’s implementation in section 3 on pg. 4). Liang teaches wherein ControlNet receives an image prompt comprising a text description of a digital image (Liang, see text prompt input in FIG. 3 on pg. 3 and control combination in FIG. 1 on pg. 1, last para on pg. 2: “Our framework supports prompt-based, stroke based, exemplar-based, and a combination of these conditions for local and global colorization”) and where the synthesized digital image depicts content corresponding to the text description (Liang, see FIGs. 1 and 3 examples attached with claim 1 – outputted images correspond to the text prompt; see prompt control section on pg. 4, section 3.2.2).
Liang teaches a method for conditioning a trained diffusion model using a text input. A person having ordinary skill in the art could have applied this method to the trained diffusion model taught by Cheng. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the text prompt input of Liang with the system of Cheng in order to provide the user with further control over the image synthesis (Liang, see text-related citations above and their applied influence demonstrated by the figures).
Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng in view of Muhammad, in further view of Liang and Yang et al. (Yang, W., Cai, F., Shu, Y., Zhang, Z., Liu, Q., & Ding, Y. (2024). Colorize at will: Harnessing diffusion prior for image colorization. IEEE Access, 12, 107287-107296.), hereinafter Yang.
Regarding claim 9 (dependent on claim 8), Cheng in view of Muhammad and Liang fails to explicitly teach wherein modifying the diffusion neural network comprises transforming, using the color control adapter, the color conditioning input according to a pixel dropping function.
However, Yang similarly teaches a method for conditioning the image colorization of a diffusion neural network (see abstract of Yang). Yang teaches wherein the method comprises: transforming, using a color control adapter, the color conditioning input according to a pixel dropping function (Yang, color hint points are sampled throughout the image and all other color values are set to grayscale, or dropped, pg. 107290, FIG. 1 caption: “On pre-processing stage, We first adopt the Simple Linear Iterative Clustering (SLIC) algorithm [41] to get superpixel image and randomly choose several grid units to generate grayscale images with color hint points as the input of colorization stage”; this is performed using the pre-processing modules, see pg. 107289, section III).
Yang teaches a method for converting strokes to hint points (Yang, see section A on pg. 107290). Utilizing color hint points in the method of Yang allows for the application of certain colors to more specific regions (Yang, pg. 107289, section III: “To represent color conditions accurately and flexibly, we introduces the innovative concept of color hint points, as first proposed by UniColor [13]. These are annotated, color-labeled points on the target grayscale image that provide explicit color information for specific regions.”). The sketch-based conditioning of Cheng’s method defines specific regions for the digital image. A person of ordinary skill in the art would recognize that color hint points could be derived from the color strokes (as performed by Yang) and used to convey color value position in the same way that the stroke conditioning does in Cheng’s method. Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the pixel dropping function of Yang with the system of Cheng in view of Muhammad and Liang in order to condition the model to more accurately apply colors to specific regions of the digital image (see last citation from Yang). Furthermore, in training the diffusion model using the sketch/color guidance (Cheng’s method) with color hint points (taught by the combination above), Cheng in view of Muhammad, Liang, and Yang teach wherein modifying the diffusion neural network comprises the transforming, as claimed.
Regarding claim 10 (dependent on claim 9), Cheng in view of Muhammad, Liang, and Yang teaches wherein transforming the color conditioning input comprises: dropping, using the color control adapter, one or more super-pixels not depicting the color value within the color conditioning input according to a first pixel dropping function (Yang, SLIC is used to generate superpixels, followed by the color hint point sampling and dropping, as described in claim 9 rejection, pg. 107290, FIG. 1 caption: “On pre-processing stage, We first adopt the Simple Linear Iterative Clustering (SLIC) algorithm [41] to get superpixel image and randomly choose several grid units to generate grayscale images with color hint points as the input of colorization stage”; see section A on pg. 107290 for more detail); or dropping, using the color control adapter, one or more super-pixels depicting the color value within the color conditioning input according to a second pixel dropping function.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng in view of Muhammad, in further view of Liang and Deng et al. (Deng, Z., Zhou, K., Wang, F., & Mi, Z. (2024). Repcontrolnet: Controlnet reparameterization. arXiv preprint arXiv:2408.09240.), hereinafter Deng.
Regarding claim 11 (dependent on claim 8), Cheng in view of Muhammad and Liang teaches a template image with a color scheme (Cheng, ccomb image of the condition preprocessing – see FIG. 2; pg. 4056, FIG. 2 caption: “At each time-step t, our proposed pipeline first performs classifier-free guidance with csketch and cstroke, which are extracted from a single input of colorful drawing ccomb”) and generating the synthesized digital image comprises generating, using the diffusion neural network, synthesized pixels according to the color scheme of the template image adapted to the image prompt (Cheng, synthesized image based on the inputted color and sketch information, see claim 8 rejection). However, Cheng in view of Muhammad and Liang fails to explicitly teach wholly wherein: receiving the color conditioning input comprises receiving a template image with a text region and a color scheme; and generating the synthesized digital image comprises generating, using the diffusion neural network, synthesized pixels according to the color scheme and the text region of the template image adapted to the image prompt (emphasis added).
However, Deng teaches a conditioning input, for a diffusion neural network, comprising receiving a template image with a text region (Deng, top left “Christmas greeting” in FIG. 2, attached below); and generating the synthesized digital image comprises generating, using the diffusion neural network, synthesized pixels according to the text region of the template image adapted to the image prompt (Deng, see generated Christmas greeting at top right of FIG. 2 on pg. 4, pg. 4, FIG. 2 caption: “Image generation results with Canny Edge conditioning. Captions: Christmas greeting with wreath and red bauble on white background, scandinavian christmas text”). A person of ordinary skill in the art would have been able to recognize that text regions could be implemented in the sketch conditioning input of Cheng, and would have been able to carry out this combination using known methods. According to Cheng method, since the sketch guides image synthesis, a text region in the conditioning sketch would result in synthesized pixels according to a text region. Doing so allows the user of Cheng invention to generate poster-like images or greetings using inputted sketch templates with text regions. Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the template image with a text region, as taught by Deng, with the sketch and color stroke conditioning inputs taught by Cheng in the system taught by Cheng in view of Muhammad and Liang in order to control the color of generated images with text regions, thus expanding the number of creative options available to the user (As shown in FIG. 2 of Deng, the sketch template image allows the user to generate both images with and without text).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng in view of Muhammad, in further view of Liang, Deng, Yang, and Chen et al. (Chen, J., Huang, Y., Lv, T., Cui, L., Chen, Q., & Wei, F. (2023). Textdiffuser: Diffusion models as text painters. Advances in Neural Information Processing Systems, 36, 9353-9387.), hereinafter Chen.
Regarding claim 12 (dependent on claim 11), Cheng in view of Muhammad, Liang, and Deng fails to explicitly teach wholly wherein generating the synthesized digital image further comprises: generating, from the template image, a super-pixel image reflecting the color scheme; determining intersected super-pixels within the super-pixel image that intersect the text region of the template image; and generating the synthesized digital image using the diffusion neural network conditioned on the intersected super-pixels.
However, Yang teaches wherein a super-pixel image reflecting the color scheme is generated from a color conditioning input is generated from an inputted stroke image (Yang, pg. 107291, 2nd to last para: “During inference phase, users are allowed to draw strokes on the grayscale image. Then, employing SLIC algorithm to get the superpixel image. If the amount of colored pixels in a cell exceeds a predefined threshold (e.g., 75% of the cell area),, we associate the color of the stroke with that color hint point.”). Doing so bounds the inputted color information to more specific areas, defined by superpixels. Cheng in view of Muhammad, Liang, and Deng discloses a base method for processing color stroke inputs, but does not specify specific methods for generating a super-pixel image reflecting the color scheme. Yang teaches a known technique of generating a super-pixel image reflecting the color scheme of an inputted image. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Yang, in the same way to the system of Cheng in view of Muhammad, Liang, and Deng and achieved predictable results of preserving structural and color boundaries of a color conditioning input, which is important for subsequent image processing operations.
Furthermore, Chen teaches a method for conditioning a diffusion model on a received template image including text regions (Chen, abstract: “TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout.”; see template layouts including text regions in FIG. 14 from pg. 24, attached below; see also FIG. 2 on pg. 4). Chen teaches further determining intersected regions within an image that intersect the text region of the template image; and generating the synthesized digital image using the diffusion neural network conditioned on the intersected regions (Chen, intersecting regions of 1) the template image and 2) a mask are utilized to output synthesized images with specified text in that region, pg. 5, section 3.3: “The user can directly start from the second stage by providing a template image (e.g., a scene image, handwritten image, or printed image), and a segmentation model is pre-trained to obtain the character-level segmentation masks (Appendix B)”; see also Appendix B).
PNG
media_image5.png
550
705
media_image5.png
Greyscale
In summary, Yang teaches wherein a super-pixel image is generated from a template image (for the purpose of color conditioning goals, as described above) and Chen teaches further utilizing that template representation (as described in the combination, based on Chen teaching receiving the template image) to condition the diffusion neural network. Chen’s conditioning is based on where the region of text is located in the template image, and therefore could be utilized with the super-pixel representation taught by Yang, regardless of whether the image is represented by super-pixels or not.
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng in view of Muhammad, in further view of Liang, Pidaparthy et al. (Pidaparthy, H., Chauhan, A., & Sudheendra, P. (2024, January). Multi-level attention aggregation for aesthetic face relighting. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 4045-4054). IEEE.), hereinafter Pidaparthy, and Mondal (Mondal, Soham. RGB to YCbCr image conversion. The Verification-blog, 10 May 2021 [online], [retrieved on 2026-08-25]. Retrieved from the Internet <URL: https://reverent-northcutt-8163cd.netlify.app/blog/design/>).
Regarding claim 13 (dependent on claim 8), Cheng in view of Muhammad and Liang fails to explicitly teach wherein the operations further comprise generating a modified color conditioning input by: converting the color conditioning input from a first color space to a second color space that defines a luminance parameter of the color conditioning input; and modifying the color conditioning input by injecting jitter into the luminance parameter.
However, Pidaparthy teaches a method for modifying image training input by injecting jitter into the luminance parameter (Pidaparthy, pg. 4061, section 4.3.2: “Further, we use several different augmentation techniques during training: 1) image flipping; 2) brightness + contrast jitter; 3) colour jitter or luminance jitter…In luminance jitter augmentation, the luminance intensity is tweaked by q”, emphasis added). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the image augmentation technique taught by Pidaparthy with the system of Cheng in view of Muhammad and Liang in order to improve the performance of the neural network (Pidaparthy, 1st para on pg. 4062: “These augmentations result in a training dataset that captures multiple different variations of the input images and thus, improves the generalization capabilities of the relighting model to vastly different data distributions.”).
The luminance value for each pixel must be calculated in order to implement luminance jitter. Accordingly, Mondal teaches a method for converting a color input from a first color space to a second color space that defines a luminance parameter of the color input (Mondal, see excerpt from the top of pg. 1 attached below).
PNG
media_image6.png
211
807
media_image6.png
Greyscale
Cheng in view of Muhammad, Liang, and Pidaparthy discloses a base method for manipulating the luminance value of pixels, but does not specify specific methods for generating the luminance value from RGB pixels. Mondal teaches a known technique of converting image values from the RGB space to the YCbCr space, where Y is the luminance component. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Mondal, in the same way to the system of Cheng in view of Muhammad, Liang, and Pidaparthy and achieved predictable results of calculating luminance values for subsequent image processing operations.
Regarding claim 14 (dependent on claim 13), Cheng in view of Muhammad, Liang, Pidaparthy, and Mondal teaches wherein the operations further comprise generating an additional synthesized digital image utilizing the diffusion neural network conditioned on the modified color conditioning input (Cheng, multiple image generation results, last para on pg. 4059: “As the input only contains rough contour and colored stroke information, our DiSS approach is capable of synthesizing multiple (i.e. multimodal) image generation results”; see FIG. 4).
PNG
media_image7.png
498
1044
media_image7.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677 /ANDREW W BEE/ Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights