DETAILED ACTION
This action is in response to the original filing on 10/14/2024. Claims 1-20 are pending and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1, 14, and 20, the claims recite “generating, by a text-to-image model, the first image based on the prompt”. It is unclear whether “based on the prompt” is intended to modify the generating or the image. For the purposes of examination, this limitation is interpreted as: generating, by a text-to-image model and based on the prompt, the first image
Regarding claims 1, 14, and 20, the claims recite “the initial slide value”. It is unclear how this limitation is intended to relate to the previously recited initial slider value. For the purposes of examination, this limitation is interpreted as: the initial slider value
Regarding claims 11 and 12, the claims recite “the set of third image”. It is unclear how this limitation is intended to relate to the previously recited set of third images. For the purposes of examination, this limitation is interpreted as: the set of third images
Regarding claims 2-13 and 15-19, claims 2-13 and 15-19 are also rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for depending on an indefinite parent claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 4-6, 14, 15, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang, Zhijie, et al. "Promptcharm: Text-to-image generation through multi-modal prompting and refinement." Proceedings of the 2024 CHI conference on human factors in computing systems (published 03/06/2024), hereinafter Wang.
Regarding claims 1, 14, and 20, Wang teaches the claim comprising:
A method, executed by a processor, comprising (Wang 4.5, We implemented and deployed PromptCharm as a webapplication. The user interface of PromptCharm is implemented with Material UI 5. The back-end of PromptCharm is based on Python Flask; All machine learning models are implemented with PyTorch 6 and Transformers 7. For the diffusion model, we use the Stable Diffusion v2-1 8. During user studies, PromptCharm ran on a server with two NVIDIA A5000 GPUs):
receiving a prompt indicative of a description of a first image to be generated; generating, by a text-to-image model, the first image based on the prompt (Wang Figure 1: PromptCharm facilitates prompt engineering in text-to-image generation with an enriched, multi-modal feedback loop. (a) Given an initial prompt from a user, PromptCharm first suggests an initial refinement based on a prompt optimization model. (b) The user can explore different styles by searching them in a large database. (c) The user can also explore similar and dissimilar image styles in a drop-down list. (d) Furthermore, the user can adjust the attention of the diffusion model to different keywords in the prompt; abs. we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user’s initial prompt; 3.2, In order to balance automation and user’s control (N2), PromptCharm provides multi-modal prompting within an interactive text box (Fig. 2 b○). In this text box, the user can Delete, Explore, or Replace one or multiple image modifiers. By exploring different modifiers, the user can know how these keywords would possibly affect the generated images without actually rendering it. If the user encounters any modifiers that they dislike, they can efficiently replace them with other similar/dissimilar modifiers in PromptCharm. This design also allows the user to explore different prompting options (N3); Figure 2: User interface of PromptCharm. (a) The user can first type their initial input prompt in a text box. PromptCharm will then automatically refine the user’s prompt. (b) PromptCharm further supports users in efficiently exploring different image styles. (c) The user can then examine the generated images with the help of model attention visualization. If they would like to refine the image, they can further (d) adjust the model’s attention to a keyword or (e) directly inpaint the image; 4.1, In PromptCharm, the user can type their own words to describe what they would like to see in their generated images in a text box (Fig. 3 a○); Alice first checks the Prompting View, where she can articulate her requirements within a designated text box (Fig. 2 a○). By clicking on the button PROMPT , PromptCharm autonomously generates a fresh prompt and presents it in another text box for her to further refine (Fig. 2 b); 5, Alice observes that this generated prompt incorporates a few additional modifiers, such as “by greg rutkowski”, “thomas kinkade”, and “trending on artstation.” She ponders how these modifications might enhance the resulting image; She ponders which modifier has contributed to the enhanced visual effects. She clicks on the modifier “thomas kinkade” and selects the Explore option; 8.2, the model should provide flexible ways to elicit user feedback. Ultimately, the goal is to create images that better align with the user’s creative intent, rather than writing complex and long prompts);
determining, based on a first language model, a set of attributes associated with the description of the first image, the set of attributes corresponding to semantics associated with the first image; generating, based on a second language model, a set of questions associated with the prompt (Wang 4.1, Automated prompt refinement. To help users refine their initial input prompts, PromptCharm leverages a state-of-the-art prompt optimization model released by Microsoft, Promptist [24]. Promptist is designed to rephrase users’ input prompts while retaining their original intentions. It is based on the GPT-2 architecture and was initially trained using a dataset that consisted of pairs of user input prompts and prompts refined by engineers. After supervised training, Promptist was then fine-tuned through reinforcement learning to optimize the prompt for generating visually pleasing images for Stable Diffusion; Suggesting popular modifiers. Previous studies have discussed the importance of modifiers (keywords that have significant effects on the generated image’s style and quality) and their impacts [36, 45]. To assist users in efficiently exploring different modifiers and image styles, PromptCharm leverages a data-mining method to extract popular modifiers from a dataset of text-to image prompts; When the user clicks on a token in the refined prompt in PromptCharm, a menu with four different options (Delete, Replace, Attention, and Explore) will pop up (Fig. 3 b○). When the user clicks on Replace, the top-3 image modifiers mined from DiffusionDB [67] that provide the most similar and dissimilar art effects will show up in a drop-down list. The user can then choose to replace the selected modifier with one of them (Fig. 3 c○). Note that the similarity between two image modifiers is calculated based on the cosine distance between their embedding obtained from the diffusion model’s text encoder; Image Style Exploration. In addition to replacing with simi lar/dissimilar image styles, the user can further explore and pick their own choices in PromptCharm (Fig. 3 e○). PromptCharm displays popular image modifiers in a drop-down list (Fig. 3 f○). The user can select from these popular modifiers, type a few new modifiers, or select from the prompt. After entering a few modifiers, the user can click on the “bulb” icon to search for images that include these modifiers from DiffusionDB [67]. The search results will be displayed in a pop-up grid. The user can then hover over a specific image of interest to check its complete text prompt. If the user is satisfied with the selected modifiers, they can click on the “add”);
generating slider boundary values and an initial slider value based on the set of questions associated with the prompt, and the first image; generating a set of sliders associated with the set of attributes, based on the slider boundary values and the initial slide value, each slider of the set of sliders being associated with a corresponding attribute of the set of attributes; receiving a user input associated with the set of sliders; generating a second image based on the user input associated with the set of sliders, and the first image; and rendering the second image (Wang Fig. 1 (d) Furthermore, the user can adjust the attention of the diffusion model to different keywords in the prompt.); 3.2, To guide users iteratively improving their creations, PromptCharm provides model explanations by visualizing the model’s attention after each iteration (N5). By observing the attention values of different words in the input prompt, the user can quickly identify if the model’s interpretation of the prompt was aligned with their intent, e.g., whether there is an important keyword the model does not pay attention to (Fig. 2 c○); 4.1, When the user clicks on Attention, a slider will pop up for the user to adjust the model’s attention to the selected keyword(s) (Fig. 3 d○). We will introduce the details of the model’s attention adjustment in Sec. 4.2. When the user clicks on Explore, the selected keyword(s) will be added to the bottom text field for further exploration (Fig. 3 e○); 4.2 PromptCharm visualizes the model’s attention in two ways: First, each token in the prompt is colored according to its saliency (Fig. 4). The higher importance a token has contributed to the generation, the darker background color it is assigned (Fig. 4 b○); for text-to-image generation, the generated image’s content could be inconsistent with the text prompt. Fig. 4 d ○shows an example of such attention misalignment. In this example, the original image (left one) misses the object “human child” in the text prompt while rendering an extra object of “wolf”. To address this, PromptCharm uses an interactive design that allows users to adjust the model’s attention to keywords in the prompt through a slider (Fig. 4 d ○). Such design is highly inspired by the recent studies on aligning human’s attention with the transformer model’s attention [26, 62, 63]. As depicted in Fig. 4 d ○, by decreasing the model’s attention to the “wolf”, the model then correctly generate an image with one “wolf” and one “human child.”);
Regarding claims 2 and 15, Wang teaches all the limitations of claim 1, further comprising:
wherein the set of questions includes at least one of: a first set of questions corresponding to slider independent questions, and a second set of questions corresponding to slider dependent questions, the generation of the slider boundary values is based on the first set of questions, and the generation of the initial slider value is based on the second set of questions (Wang Fig. 1 (d) Furthermore, the user can adjust the attention of the diffusion model to different keywords in the prompt.); 3.2, To guide users iteratively improving their creations, PromptCharm provides model explanations by visualizing the model’s attention after each iteration (N5). By observing the attention values of different words in the input prompt, the user can quickly identify if the model’s interpretation of the prompt was aligned with their intent, e.g., whether there is an important keyword the model does not pay attention to (Fig. 2 c○); 4.1, Suggesting popular modifiers. Previous studies have discussed the importance of modifiers (keywords that have significant effects on the generated image’s style and quality) and their impacts [36, 45]. To assist users in efficiently exploring different modifiers and image styles, PromptCharm leverages a data-mining method to extract popular modifiers from a dataset of text-to image prompts; When the user clicks on a token in the refined prompt in PromptCharm, a menu with four different options (Delete, Replace, Attention, and Explore) will pop up (Fig. 3 b○). When the user clicks on Replace, the top-3 image modifiers mined from DiffusionDB [67] that provide the most similar and dissimilar art effects will show up in a drop-down list. The user can then choose to replace the selected modifier with one of them (Fig. 3 c○); If the user is satisfied with the selected modifiers, they can click on the “add”; When the user clicks on Attention, a slider will pop up for the user to adjust the model’s attention to the selected keyword(s) (Fig. 3 d○). We will introduce the details of the model’s attention adjustment in Sec. 4.2. When the user clicks on Explore, the selected keyword(s) will be added to the bottom text field for further exploration (Fig. 3 e○); 4.2 PromptCharm visualizes the model’s attention in two ways: First, each token in the prompt is colored according to its saliency (Fig. 4). The higher importance a token has contributed to the generation, the darker background color it is assigned (Fig. 4 b○); 4.2, for text-to-image generation, the generated image’s content could be inconsistent with the text prompt. Fig. 4 d ○shows an example of such attention misalignment. In this example, the original image (left one) misses the object “human child” in the text prompt while rendering an extra object of “wolf”. To address this, PromptCharm uses an interactive design that allows users to adjust the model’s attention to keywords in the prompt through a slider (Fig. 4 d ○))
Regarding claim 4, Wang teaches all the limitations of claim 1, further comprising:
wherein each of the first language model and the second language model corresponds to a Large Language Model (LLM) (Wang Figure 1: PromptCharm facilitates prompt engineering in text-to-image generation with an enriched, multi-modal feedback loop. (a) Given an initial prompt from a user, PromptCharm first suggests an initial refinement based on a prompt optimization model. (b) The user can explore different styles by searching them in a large database. (c) The user can also explore similar and dissimilar image styles in a drop-down list. (d) Furthermore, the user can adjust the attention of the diffusion model to different keywords in the prompt; abs. we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user’s initial prompt; 3.2, To support N1, PromptCharm leverages a state-of-the-art model, Promptist [24], to automatically revise user’s input text prompt. After the user types their initial prompt and clicks on the PROMPT button, PromptCharm will improve the user’s prompt through re-organizing and appending suggested modifiers (Fig. 2 a○). In order to balance automation and user’s control (N2), PromptCharm provides multi-modal prompting within an interactive text box (Fig. 2 b○). In this text box, the user can Delete, Explore, or Replace one or multiple image modifiers. By exploring different modifiers, the user can know how these keywords would possibly affect the generated images without actually rendering it. If the user encounters any modifiers that they dislike, they can efficiently replace them with other similar/dissimilar modifiers in PromptCharm. This design also allows the user to explore different prompting options (N3); Figure 2: User interface of PromptCharm. (a) The user can first type their initial input prompt in a text box. PromptCharm will then automatically refine the user’s prompt. (b) PromptCharm further supports users in efficiently exploring different image styles. (c) The user can then examine the generated images with the help of model attention visualization. If they would like to refine the image, they can further (d) adjust the model’s attention to a keyword or (e) directly inpaint the image; 4.1, In PromptCharm, the user can type their own words to describe what they would like to see in their generated images in a text box (Fig. 3 a○); Alice first checks the Prompting View, where she can articulate her requirements within a designated text box (Fig. 2 a○). By clicking on the button PROMPT , PromptCharm autonomously generates a fresh prompt and presents it in another text box for her to further refine (Fig. 2 b); 5, Alice observes that this generated prompt incorporates a few additional modifiers, such as “by greg rutkowski”, “thomas kinkade”, and “trending on artstation.” She ponders how these modifications might enhance the resulting image; She ponders which modifier has contributed to the enhanced visual effects. She clicks on the modifier “thomas kinkade” and selects the Explore option; 4.2, Promptist is designed to rephrase users’ input prompts while retaining their original intentions. It is based on the GPT-2 architecture and was initially trained using a dataset that consisted of pairs of user input prompts and prompts refined by engineers; 8.2, the model should provide flexible ways to elicit user feedback. Ultimately, the goal is to create images that better align with the user’s creative intent, rather than writing complex and long prompts)
Regarding claim 5, Wang teaches all the limitations of claim 1, further comprising:
wherein the first language model is same as the second language model (Wang Figure 1: PromptCharm facilitates prompt engineering in text-to-image generation with an enriched, multi-modal feedback loop. (a) Given an initial prompt from a user, PromptCharm first suggests an initial refinement based on a prompt optimization model. (b) The user can explore different styles by searching them in a large database. (c) The user can also explore similar and dissimilar image styles in a drop-down list. (d) Furthermore, the user can adjust the attention of the diffusion model to different keywords in the prompt; abs. we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user’s initial prompt; 3.2, To support N1, PromptCharm leverages a state-of-the-art model, Promptist [24], to automatically revise user’s input text prompt. After the user types their initial prompt and clicks on the PROMPT button, PromptCharm will improve the user’s prompt through re-organizing and appending suggested modifiers (Fig. 2 a○). In order to balance automation and user’s control (N2), PromptCharm provides multi-modal prompting within an interactive text box (Fig. 2 b○). In this text box, the user can Delete, Explore, or Replace one or multiple image modifiers. By exploring different modifiers, the user can know how these keywords would possibly affect the generated images without actually rendering it. If the user encounters any modifiers that they dislike, they can efficiently replace them with other similar/dissimilar modifiers in PromptCharm. This design also allows the user to explore different prompting options (N3); Figure 2: User interface of PromptCharm. (a) The user can first type their initial input prompt in a text box. PromptCharm will then automatically refine the user’s prompt. (b) PromptCharm further supports users in efficiently exploring different image styles. (c) The user can then examine the generated images with the help of model attention visualization. If they would like to refine the image, they can further (d) adjust the model’s attention to a keyword or (e) directly inpaint the image; 4.1, In PromptCharm, the user can type their own words to describe what they would like to see in their generated images in a text box (Fig. 3 a○); Alice first checks the Prompting View, where she can articulate her requirements within a designated text box (Fig. 2 a○). By clicking on the button PROMPT , PromptCharm autonomously generates a fresh prompt and presents it in another text box for her to further refine (Fig. 2 b); 5, Alice observes that this generated prompt incorporates a few additional modifiers, such as “by greg rutkowski”, “thomas kinkade”, and “trending on artstation.” She ponders how these modifications might enhance the resulting image; She ponders which modifier has contributed to the enhanced visual effects. She clicks on the modifier “thomas kinkade” and selects the Explore option; 4.2, Promptist is designed to rephrase users’ input prompts while retaining their original intentions. It is based on the GPT-2 architecture and was initially trained using a dataset that consisted of pairs of user input prompts and prompts refined by engineers; 8.2, the model should provide flexible ways to elicit user feedback. Ultimately, the goal is to create images that better align with the user’s creative intent, rather than writing complex and long prompts)
Regarding claim 6, Wang teaches all the limitations of claim 1, further comprising:
wherein the first language model is different from the second language model (Wang Figure 1: PromptCharm facilitates prompt engineering in text-to-image generation with an enriched, multi-modal feedback loop. (a) Given an initial prompt from a user, PromptCharm first suggests an initial refinement based on a prompt optimization model. (b) The user can explore different styles by searching them in a large database. (c) The user can also explore similar and dissimilar image styles in a drop-down list. (d) Furthermore, the user can adjust the attention of the diffusion model to different keywords in the prompt; abs. we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user’s initial prompt; 3.2, To support N1, PromptCharm leverages a state-of-the-art model, Promptist [24], to automatically revise user’s input text prompt. After the user types their initial prompt and clicks on the PROMPT button, PromptCharm will improve the user’s prompt through re-organizing and appending suggested modifiers (Fig. 2 a○). In order to balance automation and user’s control (N2), PromptCharm provides multi-modal prompting within an interactive text box (Fig. 2 b○). In this text box, the user can Delete, Explore, or Replace one or multiple image modifiers. By exploring different modifiers, the user can know how these keywords would possibly affect the generated images without actually rendering it. If the user encounters any modifiers that they dislike, they can efficiently replace them with other similar/dissimilar modifiers in PromptCharm. This design also allows the user to explore different prompting options (N3); Figure 2: User interface of PromptCharm. (a) The user can first type their initial input prompt in a text box. PromptCharm will then automatically refine the user’s prompt. (b) PromptCharm further supports users in efficiently exploring different image styles. (c) The user can then examine the generated images with the help of model attention visualization. If they would like to refine the image, they can further (d) adjust the model’s attention to a keyword or (e) directly inpaint the image; 4.1, To assist users in efficiently exploring different modifiers and image styles, PromptCharm leverages a data-mining method to extract popular modifiers from a dataset of text-to image prompts; When the user clicks on a token in the refined prompt in PromptCharm, a menu with four different options (Delete, Replace, Attention, and Explore) will pop up (Fig. 3 b○). When the user clicks on Replace, the top-3 image modifiers mined from DiffusionDB [67] that provide the most similar and dissimilar art effects will show up in a drop-down list. The user can then choose to replace the selected modifier with one of them (Fig. 3 c○). Note that the similarity between two image modifiers is calculated based on the cosine distance between their embedding obtained from the diffusion model’s text encoder; 4.2, Promptist is designed to rephrase users’ input prompts while retaining their original intentions. It is based on the GPT-2 architecture and was initially trained using a dataset that consisted of pairs of user input prompts and prompts refined by engineers; 8.2, the model should provide flexible ways to elicit user feedback. Ultimately, the goal is to create images that better align with the user’s creative intent, rather than writing complex and long prompts)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Vlassis et al (US 20260017839 A1, published 01/15/2026), hereinafter Vlassis.
Regarding claim 3, Wang teaches all the limitations of claim 1. However, Wang fails to expressly disclose wherein the text-to-image model corresponds to a Generative Adversarial Network (GAN) model. In the same field of endeavor, Vlassis teaches:
wherein the text-to-image model corresponds to a Generative Adversarial Network (GAN) model (Vlassis Figs. 1-15; [0037], a user wants to generate an image of a concert having a high aesthetic quality, but does not know what to enter as a prompt beyond “a concert”. In the example, the user inputs a text generation prompt “a concert” to a user interface provided by the image generation system, makes a selection via an element of the user interface (e.g., a drop-down menu) that the image should have a high aesthetic quality, and indicates via another element of the user interface (e.g., a slider) that the image generation prompt should include eight words); [0040], some embodiments generate a synthetic image based on a system-generated prompt. Some embodiments achieve this efficiency by obtaining a text prompt and a target token indicating an attribute of the system-generated prompt, and generating the system-generated prompt using a language model that is trained to understand the target token, such that the system-generated prompt includes the attribute; [0068], the user provides the identification of the image generation prompt attribute to the user interface (for example, by providing an input to an element of the user interface, such as a drop-down menu, a slider, or the like).; [0103], image generation model 335 is trained to generate a synthetic image based on the image generation prompt; [0104], image generation model 335 comprises a generative adversarial network (GAN). In some cases, image generation model 335 comprises a diffusion model; [0105], a GAN comprises two neural networks (e.g., a generator and a discriminator) that are trained based on a contest with each other. For example, in some cases, the generator learns to generate a candidate by mapping information from a latent space to a data distribution of interest, while the discriminator distinguishes the candidate produced by the generator from a true data distribution of the data distribution of interest. In some cases, the generator's training objective is to increase an error rate of the discriminator by producing novel candidates that the discriminator classifies as “real” (e.g., belonging to the true data distribution).; [0169], a user provides an input to an element of the user interface (e.g., a drop-down element, a slider, or other graphical or text-based element) to select or otherwise indicate the image generation prompt attribute)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the text-to-image model corresponds to a Generative Adversarial Network (GAN) model as suggested in Vlassis into Wang. Doing so would be desirable because embodiments of the present disclosure improve upon conventional image generation systems by making a text-based image generation process more accurate and efficient (see Vlassis [0040]). Previous systems may result in an inefficient trial-and-error process until the user obtains a satisfactory image, or may result in the user abandoning the process before a satisfactory image is obtained (see Vlassis [0041]). Additionally, the system of Vlassis would improve the system of Wang by enabling Wang to use either Diffusion or GAN models (see Vlassis [0104]), thereby increasing the flexibility and desirability of the system by ensuring that an appropriate model is used as needed, such as using a GAN model when speed and realism are desired.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zhu et al. (US 20250123736 A1, published 04/17/2025), hereinafter Zhu.
Regarding claim 7, Wang teaches all the limitations of claim 1. However, Wang fails to expressly disclose wherein the set of attributes correspond to at least one of: an age of a person, a hair color of the person, facial expressions of the person, a body-built type of the person, a height of the person, a direction of a face of the person, or a gender of the person. In the same field of endeavor, Zhu teaches:
wherein the set of attributes correspond to at least one of: an age of a person, a hair color of the person, facial expressions of the person, a body-built type of the person, a height of the person, a direction of a face of the person, or a gender of the person (Zhu Figs. 1-10; [0029], In FIG. 3A, natural language input 308 is entered into prompt area 302 and contains the words “Dog and Cat are fighting.”; FIG. 3B shows the same user interface and the same natural language input 308, as in FIG. 3A. However, in the example of FIG. 3B, the user has selected (e.g. highlighted, clicked, etc.) “Dog” and “Cat” in the prompt area and then, by interacting with button 314 (Add), populated the user interface control with only two sliders: namely, slider user interface components 316a and 316b. As seen in FIG. 3B, slider user interface components 316a and 316b are sliders associated with each of the selected words, “Dog” and “Cat.”; [0036], the user input is received in user interface control area 304 as the user interacts with user interface 300 to independently change the position of the slider buttons for the various sliders; [0037], By changing the slider button positions to the extreme bottom and extreme top positions, in this example, the user is instructing the system that the preview must be changed to de-emphasize (i.e. completely remove) “Dog” elements and simultaneously emphasize “Cat” elements seen in the preview. Accordingly, the preview for output image 320 seen in FIG. 3C includes two cats, instead of one cat, as seen in output image 314 of FIG. 3A, but does not include any dogs.; [0043], the user may move the slider button to the bottom position to indicate that the associated word should be completely removed from the output image, whereas the top position indicates the associated word should be emphasized in the output image, appear more frequently in the output, appear larger in the output, or any combination thereof; [0081], FIG. 8D illustrates another content control technique where the elements are associated with coordinates in sliders in the UI to alter the attribute values. In this example, a user enters the text “A business man is entering the office.” Content elements are each associated with each word and a user may adjust attributes using sliders. At 807, the sliders are all set to medium positions. Thus, the result is uniform at in 801. However, at 808, the sliders may be adjusted to increase some content element attribute values and decrease other content element attribute values. In this example, “business” is increased and “office” is decreased. Accordingly, the image shows a man in a business suit with a commercial office building in the background, but not an office conference room)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the set of attributes correspond to at least one of: an age of a person, a hair color of the person, facial expressions of the person, a body-built type of the person, a height of the person, a direction of a face of the person, or a gender of the person as suggested in Zhu into Wang. Doing so would be desirable because content generation involves generating images, video, and other forms of content. The design process is typically very creative. However, modern content generation systems are digital platforms that must adhere to the constraints imposed by computers and computer programming. Content workers are typically skilled creative artisans, but such users often may not be skilled in the technical nuances of computer programming. Accordingly, there is a tension between the digital world of bits, bytes, and technical computer code and the creative world of skilled artisans. Indeed, the technicalities of computer code can often limit or constrain the creative process. Thus, it is a challenge to free creative artisans from the rigid structure of computer code and programming (see Zhu [0003]). Businesses and other enterprises often dedicate a large number of resources to develop vast amounts of references, content, and the like for different purposes. For example, a business may create training videos to teach new employees vital aspects of the job. However, the time and monetary cost necessary to produce high quality content may be untenable for some departments. The use of artificial intelligence for generative content creation can significantly reduce the time and cost to develop these materials. In the past, however, only scientists, engineers, and skilled researchers possessed the necessary expertise to develop and leverage artificial intelligence for generative content creation. Accordingly, there is a growing need for intuitive tools capable of controlling artificial intelligence to automate generative content creation, and thus, reduce the amount of resources needed to produce high quality content (see Zhu [0016]). Features and advantages of such configurable visual representations include providing users flexible control over the subsequent processing of each word of the natural language input (see Zhu [0021]). Additionally, the system of Zhu would improve the system of Wang by further clarifying that sliders can be provided a larger array Wang’s keywords related to age and gender (such as highlighted keyword “child” in Wang Fig. 1, 2).
Claims 8, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Upchurch (US 11334147 B1, published 05/17/2022).
Regarding claims 8 and 16, Wang teaches all the limitations of claim 1, further comprising:
wherein the generation of the slider boundary values correspond to a first model, and the generation of the initial slider value corresponds to a second model (Wang Fig. 1 (d) Furthermore, the user can adjust the attention of the diffusion model to different keywords in the prompt.); 3.2, To guide users iteratively improving their creations, PromptCharm provides model explanations by visualizing the model’s attention after each iteration (N5). By observing the attention values of different words in the input prompt, the user can quickly identify if the model’s interpretation of the prompt was aligned with their intent, e.g., whether there is an important keyword the model does not pay attention to (Fig. 2 c○); 4.1, Suggesting popular modifiers. Previous studies have discussed the importance of modifiers (keywords that have significant effects on the generated image’s style and quality) and their impacts [36, 45]. To assist users in efficiently exploring different modifiers and image styles, PromptCharm leverages a data-mining method to extract popular modifiers from a dataset of text-to image prompts; When the user clicks on a token in the refined prompt in PromptCharm, a menu with four different options (Delete, Replace, Attention, and Explore) will pop up (Fig. 3 b○). When the user clicks on Replace, the top-3 image modifiers mined from DiffusionDB [67] that provide the most similar and dissimilar art effects will show up in a drop-down list. The user can then choose to replace the selected modifier with one of them (Fig. 3 c○); If the user is satisfied with the selected modifiers, they can click on the “add”; When the user clicks on Attention, a slider will pop up for the user to adjust the model’s attention to the selected keyword(s) (Fig. 3 d○). We will introduce the details of the model’s attention adjustment in Sec. 4.2. When the user clicks on Explore, the selected keyword(s) will be added to the bottom text field for further exploration (Fig. 3 e○); 4.2 PromptCharm visualizes the model’s attention in two ways: First, each token in the prompt is colored according to its saliency (Fig. 4). The higher importance a token has contributed to the generation, the darker background color it is assigned (Fig. 4 b○); 4.2, for text-to-image generation, the generated image’s content could be inconsistent with the text prompt. Fig. 4 d ○shows an example of such attention misalignment. In this example, the original image (left one) misses the object “human child” in the text prompt while rendering an extra object of “wolf”. To address this, PromptCharm uses an interactive design that allows users to adjust the model’s attention to keywords in the prompt through a slider (Fig. 4 d ○)); 4.1, To assist users in efficiently exploring different modifiers and image styles, PromptCharm leverages a data-mining method to extract popular modifiers from a dataset of text-to image prompts; When the user clicks on a token in the refined prompt in PromptCharm, a menu with four different options (Delete, Replace, Attention, and Explore) will pop up (Fig. 3 b○). When the user clicks on Replace, the top-3 image modifiers mined from DiffusionDB [67] that provide the most similar and dissimilar art effects will show up in a drop-down list. The user can then choose to replace the selected modifier with one of them (Fig. 3 c○). Note that the similarity between two image modifiers is calculated based on the cosine distance between their embedding obtained from the diffusion model’s text encoder; 4.2, Promptist is designed to rephrase users’ input prompts while retaining their original intentions. It is based on the GPT-2 architecture and was initially trained using a dataset that consisted of pairs of user input prompts and prompts refined by engineers)
However, Wang fails to expressly disclose wherein the generation of the slider boundary values correspond to a first visual question answering (VQA) model, and the generation of the initial slider value corresponds to a second VQA model. In the same field of endeavor, Upchurch teaches:
wherein the generation of the slider boundary values correspond to a first visual question answering (VQA) model, and the generation of the initial slider value corresponds to a second VQA model (Upchurch Figs. 1-10; col. 1 [line 13], The present disclosure generally relates to a visual question and answer (VQA) technique, and in particular, to systems, methods, and devices for training a machine learning (ML) system to apply virtual effects using a training dataset generated based on VQA data; col. 2 [line 27], a runtime method described herein enables a trained virtual effect system to identify a portion of an image associated with a virtual action and apply the virtual effects to the portion of the image during runtime; col. 9 [line 3], the VQA presenter 248 includes a dynamic updater 249, which is configured to dynamically update the reference image based on the VQA answers to the VQA questions from the user; col. 10 [line 9], the input handler 250 is configured to obtain (e.g., receive, retrieve, or detect) user inputs that correspond to the VQA answers from the user to the VQA questions. For example, the user inputs correspond to voice commands, gestural inputs, eye tracking inputs, limb/hand tracking inputs, and/or the like. As another example, the user inputs correspond to selection and/or manipulation of various affordances such as radio buttons, sliders, and/or the like; col. 15 [line 20], As shown in FIG. 4A, while generating the training dataset 255, the virtual effect system presents a reference image 402 to a user with an accompanying object mask 404 that highlights or surrounds a wood pile within the reference image 402. As shown in FIG. 4A, while generating the training dataset 255, the virtual effect system presents an annotation prompt window 425 in concert with the reference image 402; col. 15 [line 32], As shown in FIG. 4A, the annotation prompt window 425 is associated with a virtual action that corresponds to applying a flame applicator to the object mask 404 (e.g., the wood pile). In this example, the flame applicator virtual action is associated with a plurality of virtual effect instructions: one or more visual flames, a visual glow, and an audible snap and crackle. As such, the annotation prompt window 425 includes VQA questions associated with the plurality of virtual effect instructions including a first radio button 410 indicating a binary VQA question as to whether or not the wood pile should exhibit one or more visual flames when the virtual action is applied thereto, a second radio button 420 indicating a binary VQA question as to whether or not the wood pile should exhibit a visual glow when the virtual action is applied thereto, and a third radio button 430 indicating a binary VQA question as to whether or not an audible snap and crackle should occur when the virtual action is applied thereto. As shown in FIG. 4A, the first radio button 410 is currently set to the “on” position, the second radio button 420 is currently set to the “off” position, and the third radio button 430 is currently set to the “on” position; col. 15 [line 53], As shown in FIG. 4A, the annotation prompt window 425 includes a user manipulatable slider 412 associated with the intensity of the visual flames and a user manipulatable text entry field 414 associated with the number of visual flames. As shown in FIG. 4A, the user manipulatable slider 412 is currently set to a middle position, and the text entry field 414 includes the number “3”; col. 15 [line 60], As shown in FIG. 4A, the annotation prompt window 425 also includes a user manipulatable slider 422 associated with the intensity of the visual glow and a user manipulatable slider 432 associated with the intensity of the audible snap and crackle. As shown in FIG. 4A, the user manipulatable slider 422 is currently set to a middle position, and the user manipulatable slider 432 is also currently set to a middle position; col. 20 [line 8], As shown in FIG. 7B, the VFX processor 280 obtains (e.g., receives, retrieves, or determines/generates) one or more virtual effect visualizations based on the estimated VQA answers 760 from the ML system 260 (or the normalized values or the like thereof) and a virtual content library 282. As shown in FIG. 7B, the rendering and compositing engine 296 composites the image stream 708 of the physical environment 105 with the one or more virtual effect visualizations in order to generate a modified environment 297 that includes the one or more virtual effect visualizations; col. 21 [line 35] s shown in FIG. 8D, during the instance 840 (e.g., associated with time T.sub.4) of the virtual effect application scenario, the electronic device 120 applies the flame applicator virtual effect to the couch 802 in response to detecting the user input 839 in FIG. 8D. In FIG. 8D, the application of the flame applicator virtual action to the couch 802 causes visual flames 844 to be overlaid on the couch 802 within the XR environment 128 and also causes an audible snap and crackle 846 to accompany the visual flames 844; col. 22 [line 29], the ML system 260 ingests the reference image 510 along with the object mask 508, the virtual action 502, and the one or more VQA questions 506 and outputs one or more estimated VQA answers 602. For example, with reference to FIG. 6, the at least one estimated material property response corresponds to the estimated VQAAs 602; col. 22 [line 45], the reference image corresponds to a photorealistic/synthetic image)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the generation of the slider boundary values correspond to a first visual question answering (VQA) model, and the generation of the initial slider value corresponds to a second VQA model as suggested in Upchurch into Wang. Doing so would be desirable a user viewing a fireplace may initiate a virtual action to light a fireplace and expect one or more accompanying virtual effects such as a visual glow, visual flames, visual smoke, crackling sounds, and/or the like. Current systems lack logic to present virtual effects to a user interacting with a representation, and current systems also lack logic to present virtual effects to a user in real-time while the user interacts with the representation (see Upchurch col. 1 [line 22]). A runtime method described herein enables a trained virtual effect system to identify a portion of an image associated with a virtual action and apply the virtual effects to the portion of the image during runtime (see Upchurch col. 2 [line 27]). Additionally, the system of Upchurch would improve the system of Wang by providing efficient and effective methods of determining desired attributes associated with easy-to-understand, flexible input elicitations (see Wang 8.2) and easy-to-use input mechanisms, such as sliders (see Upchurch Fig. 4).
Regarding claim 9, Wang in view of Upchurch teaches all the limitations of claim 8. Upchurch further teaches:
wherein the first VQA model is same as the second VQA model (Upchurch Figs. 1-10; col. 1 [line 13], The present disclosure generally relates to a visual question and answer (VQA) technique, and in particular, to systems, methods, and devices for training a machine learning (ML) system to apply virtual effects using a training dataset generated based on VQA data; col. 2 [line 27], a runtime method described herein enables a trained virtual effect system to identify a portion of an image associated with a virtual action and apply the virtual effects to the portion of the image during runtime; col. 9 [line 3], the VQA presenter 248 includes a dynamic updater 249, which is configured to dynamically update the reference image based on the VQA answers to the VQA questions from the user; col. 10 [line 9], the input handler 250 is configured to obtain (e.g., receive, retrieve, or detect) user inputs that correspond to the VQA answers from the user to the VQA questions. For example, the user inputs correspond to voice commands, gestural inputs, eye tracking inputs, limb/hand tracking inputs, and/or the like. As another example, the user inputs correspond to selection and/or manipulation of various affordances such as radio buttons, sliders, and/or the like; col. 15 [line 20], As shown in FIG. 4A, while generating the training dataset 255, the virtual effect system presents a reference image 402 to a user with an accompanying object mask 404 that highlights or surrounds a wood pile within the reference image 402. As shown in FIG. 4A, while generating the training dataset 255, the virtual effect system presents an annotation prompt window 425 in concert with the reference image 402; col. 15 [line 32], As shown in FIG. 4A, the annotation prompt window 425 is associated with a virtual action that corresponds to applying a flame applicator to the object mask 404 (e.g., the wood pile). In this example, the flame applicator virtual action is associated with a plurality of virtual effect instructions: one or more visual flames, a visual glow, and an audible snap and crackle. As such, the annotation prompt window 425 includes VQA questions associated with the plurality of virtual effect instructions including a first radio button 410 indicating a binary VQA question as to whether or not the wood pile should exhibit one or more visual flames when the virtual action is applied thereto, a second radio button 420 indicating a binary VQA question as to whether or not the wood pile should exhibit a visual glow when the virtual action is applied thereto, and a third radio button 430 indicating a binary VQA question as to whether or not an audible snap and crackle should occur when the virtual action is applied thereto. As shown in FIG. 4A, the first radio button 410 is currently set to the “on” position, the second radio button 420 is currently set to the “off” position, and the third radio button 430 is currently set to the “on” position; col. 15 [line 53], As shown in FIG. 4A, the annotation prompt window 425 includes a user manipulatable slider 412 associated with the intensity of the visual flames and a user manipulatable text entry field 414 associated with the number of visual flames. As shown in FIG. 4A, the user manipulatable slider 412 is currently set to a middle position, and the text entry field 414 includes the number “3”; col. 15 [line 60], As shown in FIG. 4A, the annotation prompt window 425 also includes a user manipulatable slider 422 associated with the intensity of the visual glow and a user manipulatable slider 432 associated with the intensity of the audible snap and crackle. As shown in FIG. 4A, the user manipulatable slider 422 is currently set to a middle position, and the user manipulatable slider 432 is also currently set to a middle position; col. 20 [line 8], As shown in FIG. 7B, the VFX processor 280 obtains (e.g., receives, retrieves, or determines/generates) one or more virtual effect visualizations based on the estimated VQA answers 760 from the ML system 260 (or the normalized values or the like thereof) and a virtual content library 282. As shown in FIG. 7B, the rendering and compositing engine 296 composites the image stream 708 of the physical environment 105 with the one or more virtual effect visualizations in order to generate a modified environment 297 that includes the one or more virtual effect visualizations; col. 21 [line 35] s shown in FIG. 8D, during the instance 840 (e.g., associated with time T.sub.4) of the virtual effect application scenario, the electronic device 120 applies the flame applicator virtual effect to the couch 802 in response to detecting the user input 839 in FIG. 8D. In FIG. 8D, the application of the flame applicator virtual action to the couch 802 causes visual flames 844 to be overlaid on the couch 802 within the XR environment 128 and also causes an audible snap and crackle 846 to accompany the visual flames 844; col. 22 [line 29], the ML system 260 ingests the reference image 510 along with the object mask 508, the virtual action 502, and the one or more VQA questions 506 and outputs one or more estimated VQA answers 602. For example, with reference to FIG. 6, the at least one estimated material property response corresponds to the estimated VQAAs 602; col. 22 [line 45], the reference image corresponds to a photorealistic/synthetic image)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the first VQA model is same as the second VQA model as suggested in Upchurch into Wang. Doing so would be desirable a user viewing a fireplace may initiate a virtual action to light a fireplace and expect one or more accompanying virtual effects such as a visual glow, visual flames, visual smoke, crackling sounds, and/or the like. Current systems lack logic to present virtual effects to a user interacting with a representation, and current systems also lack logic to present virtual effects to a user in real-time while the user interacts with the representation (see Upchurch col. 1 [line 22]). a runtime method described herein enables a trained virtual effect system to identify a portion of an image associated with a virtual action and apply the virtual effects to the portion of the image during runtime (see Upchurch col. 2 [line 27]). Additionally, the system of Upchurch would improve the system of Wang by providing efficient and effective methods of determining desired attributes associated with easy-to-understand, flexible input elicitations (see Wang 8.2) and easy-to-use input mechanisms, such as sliders (see Upchurch Fig. 4).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Upchurch in further view of Hu, Yushi, et al. "Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering." 2023 ieee/cvf international conference on computer vision (iccv). IEEE, 2023 (published 08/17/2023).
Regarding claim 10, Wang in view of Upchurch teaches all the limitations of claim 8. However, Wang in view of Upchurch fails to expressly disclose wherein the first VQA model is different from the second VQA model. In the same field of endeavor, Hu teaches:
wherein the first VQA model is different from the second VQA model (Hu abstract, We introduce TIFA (Text-to-Image Faithfulness evaluation with question Answering), an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA). Specifically, given a text input, we automatically generate several question-answer pairs using a language model. We calculate image faithfulness by checking whether existing VQA models can answer these questions using the generated image; Fig. 2; Figure 2. (a) Overview of how TIFA evaluates the faithfulness of a synthesized image. TIFA uses a language model (LM), a question answering (QA) model, and a visual-question-answering (VQA) model. Given a text input, we generate several question-answer pairs with the LM and then filter them via the QA model. To evaluate the faithfulness of a synthesized image to the text input, a VQA model answers these visual questions using the image, and we check the answers for correctness. (b) TIFA v1.0 benchmark; 1, To evaluate a generated image, we use a visual-question-answering (VQA) system (here, mPLUG large [31], BLIP-2 [32], etc.); 3.2, To ensure the quality of generated images, we use UnifiedQA [25] to verify the GPT-3 generated question-answer pairs and filter out the ones that GPT-3 and UnifiedQA do not agree on. UnifiedQA2 is a state-of-the-art multi-task question-answering model that can answer both multiple choice and free-form questions; 3.3, Since our questions contain a diverse set of visual elements (e.g., activity, art style), we use open-domain pre-trained vision-language models as our VQA model (rather than closed-class classification models fine-tuned on VQAv2[14]). We provide tools to easily perform VQA on arbitrary images and questions, based on 5 state-of-the-art VQA models trained with distinct data and strategies; 4, The benchmark also comes with Python pip-installable APIs to perform VQA with various state-of-the-art VQA models on arbitrary visual questions. The overall TIFA for each text-to-image model is computed by averaging TIFA scores of images generated from each text input in the benchmark; 6, The VQA models used within the framework)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the first VQA model is different from the second VQA model as suggested in Hu into Wang. Doing so would be desirable because TIFA is a reference-free metric that allows for fine-grained and interpretable evaluations of generated images. TIFA also has better correlations with human judgments than existing metrics (see Hu abstract). A critical bottleneck, therefore, is the lack of reliable automatic evaluation metrics for text-to-image generation faithfulness. One of the popular metrics is CLIPScore [17], which measures the cosine similarity between the CLIP embeddings [42] of the text input and the generated image. However, since CLIP is not effective at counting objects [42], or reasoning compositionally [37], CLIPScore is unreliable and often inaccurate (see Hu 1). TIFA is applicable to any text prompt (see Hu Figure 2). TIFA exploits the power of the language models to decompose the text input into fine-grained probes, which allows VQA to capture more nuanced aspects of the text input and the generated image (see Hu 6).
Allowable Subject Matter
Claims 11 and 17 are rejected under 35 U.S.C. 112(b) but would be allowable if the rejection were overcome and the claims rewritten in independent form including all of the limitations of the base claims and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter:
Claim 11 includes all limitations of parent claims 1 and 8. Claim 17 includes all limitations of parent claims 14 and 16. The prior arts of record when taken individually or in combination do not expressly teach or render obvious the limitations recited in claims 11 and 17, when taken in the context of the claims as a whole. Claims 12, 13, 18, and 19 further include all limitations of parent claims 1, 8, 11, 14, 16, and 17.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Rowe (US 20250348788 A1) Figs. 1-18 and [0031], [0056], [0068-0072].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN T REPSHER III whose telephone number is (571)272-7487. The examiner can normally be reached Monday - Friday, 8AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN T REPSHER III/ Primary Examiner, Art Unit 2143
Prosecution Insights