DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 3, 8, 10, 15 and 17 are objected to because of the following informalities:
For claim 1, Examiner believes this claim should be amended in the following manner:
A method comprising:
receiving, within a text entry interface element of an image generation system, a partial generative image prompt from a user;
selecting, by the image generation system, a set of text prompt suggestions to present to the user based on the partial generative image prompt, each text prompt suggestion corresponding to a characteristic of an image to be generated based on the partial generative image prompt;
receiving, by the image generation system, a selection of a text prompt suggestion by the user;
modifying, by the image generation system, the partial generative image prompt using the selected text prompt suggestion to produce a modified generative image prompt;
applying, by the image generation system, a generative image model to the modified generative image prompt to produce a set of images; and
displaying, by the image generation system, the
For claim 3, Examiner believes this claim should be amended in the following manner:
The method of claim 2, wherein the previously generated set of images are provided as an input to the generative image model
For claim 8, Examiner believes this claim should be amended in the following manner:
A non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor, cause the hardware processor to perform steps comprising:
receiving, within a text entry interface element of an image generation system, a partial generative image prompt from a user;
selecting, by the image generation system, a set of text prompt suggestions to present to the user based on the partial generative image prompt, each text prompt suggestion corresponding to a characteristic of an image to be generated based on the partial generative image prompt;
receiving, by the image generation system, a selection of a text prompt suggestion by the user;
modifying, by the image generation system, the partial generative image prompt using the selected text prompt suggestion to produce a modified generative image prompt;
applying, by the image generation system, a generative image model to the modified generative image prompt to produce a set of images; and
displaying, by the image generation system, the
For claim 10, Examiner believes this claim should be amended in the following manner:
The non-transitory computer-readable storage medium of claim 9, wherein the previously generated set of images are provided as an input to the generative image model
For claim 15, Examiner believes this claim should be amended in the following manner:
A system comprising:
a hardware processor; and
a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor, cause the hardware processor to perform steps comprising:
receiving, within a text entry interface element of an image generation system, a partial generative image prompt from a user;
selecting, by the image generation system, a set of text prompt suggestions to present to the user based on the partial generative image prompt, each text prompt suggestion corresponding to a characteristic of an image to be generated based on the partial generative image prompt;
receiving, by the image generation system, a selection of a text prompt suggestion by the user;
modifying, by the image generation system, the partial generative image prompt using the selected text prompt suggestion to produce a modified generative image prompt;
applying, by the image generation system, a generative image model to the modified generative image prompt to produce a set of images; and
displaying, by the image generation system, the
For claim 17, Examiner believes this claim should be amended in the following manner:
The system of claim 16, wherein the previously generated set of images are provided as an input to the generative image model.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 10 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
For dependent claim 3, parent claim 1 establishes “a set of images” and “generated set of images” and parent claim 2 establishes a “previously generated set of images”. Claim 3 goes on to recite the phrase “the set of images” and it is unclear and ambiguous to which of the previously established “set of images”, “generated set of images” and “previously generated set of images” is being referenced by the phrase “the set of images”. Examiner has suggested amendments in the claim objections discussed above the resolve the ambiguities.
For dependent claim 10, parent claim 8 establishes “a set of images” and “generated set of images” and parent claim 9 establishes a “previously generated set of images”. Claim 10 goes on to recite the phrase “the set of images” and it is unclear and ambiguous to which of the previously established “set of images”, “generated set of images” and “previously generated set of images” is being referenced by the phrase “the set of images”. Examiner has suggested amendments in the claim objections discussed above the resolve the ambiguities.
For dependent claim 17, parent claim 15 establishes “a set of images” and “generated set of images” and parent claim 16 establishes a “previously generated set of images”. Claim 17 goes on to recite the phrase “the set of images” and it is unclear and ambiguous to which of the previously established “set of images”, “generated set of images” and “previously generated set of images” is being referenced by the phrase “the set of images”. Examiner has suggested amendments in the claim objections discussed above the resolve the ambiguities.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-6, 8, 11-13, 15 and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng et al., PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation, arXiv, August 2023 (hereinafter “Feng”) in view of Brade et al, Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models, arXiv, April 2023 (hereinafter “Brade”) and Bean (U.S. Patent Application Publication 2024/0320867 A1).
For claim 1, Feng discloses a method (pages 1 and 9) comprising: receiving, within a text entry interface element of an image generation system, a partial generative image prompt from a user (disclosing an image generation system (page 4) for presenting a user interface with a text entry interface element to receive a first prompt as a partial generative image prompt from a user (page 6/Fig. 4)); selecting, by the image generation system, a set of text prompt suggestions to present to the user based on the partial generative image prompt, each text prompt suggestion corresponding to a characteristic of an image to be generated based on the partial generative image prompt (disclosing the image generation system selects text prompt suggestions for presentation to the user based on the first prompt where each suggestion corresponds to a characteristic of an image to be generated based on the first prompt (pages 6-7/Fig. 4)); receiving, by the image generation system, a selection of a text prompt suggestion by the user (disclosing the image generation system receives a selection of a text prompt suggestion from the user (pages 6-7/Fig. 4)); modifying, by the image generation system, the partial generative image prompt using the selected text prompt suggestion to produce a modified generative image prompt (disclosing the image generation system modifies the first prompt using the selected text prompt suggestion to produce a modified generative image prompt (pages 6-7/Fig. 4)); applying, by the image generation system, a generative image model to the modified generative image prompt to produce a set of images (disclosing the image generation system applies a generative image model such as a Stable Diffusion model to the modified generative image prompt to produce a set of images (pages 6-7/Fig. 4)); and displaying, by the image generation system, the generated set of images (disclosing the image generation system displays the set of images within the user interface to the user (pages 6-7/Fig. 4)).
Feng does not specifically disclose displaying generated images within a digital canvas.
However, these limitations are well-known in the art as disclosed in Brade.
Brade similarly discloses a method of automatic prompt suggestion to assist in image generation (page 1). Brade explains it is known to display generated images on a digital canvas (pages 6-7 and 10). It follows Feng may be accordingly modified with the teachings of Brade to display its generated set of images on a digital canvas.
A person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention would find it obvious to modify Feng with the teachings of Brade. Brade is analogous art in dealing with a method of automatic prompt suggestion to assist in image generation (page 1). Brade discloses its use of a digital canvas is advantageous in appropriately visualizing generated images to a user for interaction (pages 6-7 and 10). Consequently, a PHOSITA would incorporate the teachings of Brade into Feng for appropriately visualizing generated images to a user for interaction.
Examiner finds Feng and Brade discloses an image generation system for the reasons discussed above.
In any case, these limitations are well-known in the art as disclosed in Bean.
Bean similarly discloses a method of prompt suggestion to assist in image generation (par. 22). Bean explains its method may be implemented with a computer system to perform image generation as an image generation system (Fig. 9; par. 22 and 95). It follows Feng and Brade may be accordingly modified with the teachings of Bean to implement its method with an image generation system.
A PHOSITA before the effective filing date of the claimed invention would find it obvious to modify Feng and Brade with the teachings of Bean. Bean is analogous art in dealing with a method of prompt suggestion to assist in image generation (par. 22). Bean discloses its use of image generation system is advantageous in appropriately implementing steps to carry out a method to perform image generation (Fig. 9; par. 22 and 95). Consequently, a PHOSITA would incorporate the teachings of Bean into Feng and Brade for appropriately implementing steps to carry out a method to perform image generation. Therefore, claim 1 is rendered obvious to a PHOSITA before the effective filing date of the claimed invention.
For claim 4, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of images includes both subject matter from the partial generative image prompt and subject matter from the selected text prompt suggestion (Feng discloses the set of images includes subject matter from the first prompt as the partial generative image prompt and subject matter from the selected text prompt suggestion (page 6/Fig. 4)).
For claim 5, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of text prompt suggestions are selected based additionally on other images generated by the user within the digital canvas (Feng discloses the set of text prompt suggestions are selected based on images generated by the user from the first prompt (page 6/Fig. 4); Brade similarly discloses a method of automatic prompt suggestion to assist in image generation (page 1); Brade explains it is known to display generated images on a digital canvas (pages 6-7 and 10); and it follows Feng may be accordingly modified with the teachings of Brade to present images generated from its first prompt on a digital canvas).
For claim 6, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of text prompt suggestions are selected based additionally on other images generated by the user during a same working session (Feng discloses the set of text prompt suggestions are selected based on images generated by the user from the first prompt during a same usage scenario as a same working session (page 6/Fig. 4)).
For claim 8, Feng as modified by Brade and Bean discloses a non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor (Feng discloses a method (pages 1 and 9); Bean similarly discloses a method of prompt suggestion to assist in image generation (par. 22); Bean explains its method may be implemented with a computer system to perform image generation as an image generation system (Fig. 9; par. 22 and 95); Bean explains the computer system includes memory for storing instructions for execution by a processor to perform the functions of the computer system (par. 95 and 104); and it follows Feng and Brade may be accordingly modified with the teachings of Bean to implement its method with an image generation system), cause the hardware processor to perform steps of the method of claim 1 (see above as to claim 1).
For claim 11, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 4. It follows claim 11 is rejected for the same reasons as to claim 8 and claim 4.
For claim 12, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 5. It follows claim 12 is rejected for the same reasons as to claim 8 and claim 5.
For claim 13, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 6. It follows claim 13 is rejected for the same reasons as to claim 8 and claim 6.
For claim 15, Feng as modified by Brade and Bean discloses a system comprising: a hardware processor; and a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor (Feng discloses a method (pages 1 and 9); Bean similarly discloses a method of prompt suggestion to assist in image generation (par. 22); Bean explains its method may be implemented with a computer system to perform image generation as an image generation system (Fig. 9; par. 22 and 95); Bean explains the computer system includes memory for storing instructions for execution by a processor to perform the functions of the computer system (par. 95 and 104); and it follows Feng and Brade may be accordingly modified with the teachings of Bean to implement its method with an image generation system), cause the hardware processor to perform steps of the method of claim 1 (see above as to claim 1).
For claim 18, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 4. It follows claim 18 is rejected for the same reasons as to claim 15 and claim 4.
For claim 19, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 5. It follows claim 19 is rejected for the same reasons as to claim 15 and claim 5.
For claim 20, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 6. It follows claim 20 is rejected for the same reasons as to claim 15 and claim 6.
Claim(s) 2, 3, 9, 10, 16 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng in view of Brade and Bean further in view of Zhang et al., SINE: SINgle Image Editing with Text-to-Image Diffusion Models, arXiv, December 2022 (hereinafter “Zhang”).
For claim 2, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of images are generated by modifying a previously generated set of images generated based on the partial generative image prompt from a user (Feng discloses previously generated images based on the first prompt as the partial generative image prompt are modified by the selected text prompt suggestion and the modified generative image prompt to generate the set of images based on the selected text prompt suggestion (pages 6-7/Fig. 4)).
Feng as modified by Brade and Bean does not specifically disclose images are modified through input to a generative image model.
However, these limitations are well-known in the art as disclosed in Zhang.
Zhang similarly discloses a method for image generation with a generative image model (page 1). Zhang explains an image may be input into the image generation model for modification by image editing in accordance with a text prompt (page 1/Fig. 1 and page 5/Fig. 3). It follows Feng, Brade and Bean may be accordingly modified with the teachings of Zhang to provide its previously generated images as input into its generative image model for generating its set of images based on its modified generative image prompt.
A PHOSITA before the effective filing date of the claimed invention would find it obvious to modify Feng, Brade and Bean with the teachings of Zhang. Zhang is analogous art in dealing with a method for image generation with a generative image model (page 1). Zhang discloses its use of image editing is advantageous in appropriately modifying an image input into a generative image model through guidance from a text prompt (page 1/Fig. 1 and page 5/Fig. 3). Consequently, a PHOSITA would incorporate the teachings of Zhang into Feng, Brade and Bean for appropriately modifying an image input into a generative image model through guidance from a text prompt. Therefore, claim 2 is rendered obvious to a PHOSITA before the effective filing date of the claimed invention.
For claim 3, depending on claim 2, Feng as modified by Brade, Bean and Zhang discloses wherein the previously generated set of images are provided as an input to the generative image model to produce the set of images (Feng discloses previously generated images based on the first prompt as the partial generative image prompt are modified by the selected text prompt suggestion and the modified generative image prompt to generate the set of images based on the selected text prompt suggestion (pages 6-7/Fig. 4); Zhang similarly discloses a method for image generation with a generative image model (page 1); Zhang explains an image may be input into the image generation model for modification by image editing in accordance with a text prompt (page 1/Fig. 1 and page 5/Fig. 3); and it follows Feng, Brade and Bean may be accordingly modified with the teachings of Zhang to provide its previously generated images as input into its generative image model for generating its set of images based on its modified generative image prompt).
For claim 9, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 2. It follows claim 9 is rejected for the same reasons as to claim 8 and claim 2.
For claim 10, depending on claim 9, this claim is a combination of the limitations of claim 9 and claim 3. It follows claim 10 is rejected for the same reasons as to claim 9 and claim 3.
For claim 16, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 2. It follows claim 16 is rejected for the same reasons as to claim 15 and claim 2.
For claim 17, depending on claim 16, this claim is a combination of the limitations of claim 16 and claim 3. It follows claim 17 is rejected for the same reasons as to claim 16 and claim 3.
Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng in view of Brade and Bean further in view of Wang et al., DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models, arXiv, July 2023 (hereinafter “Wang”).
For claim 7, depending on claim 1, Feng as modified by Brade and Bean does not disclose a set of text prompt suggestions are selected based on images and text prompts used by other users.
However, these limitations are well-known in the art as disclosed in Wang.
Wang similarly discloses a method for prompt suggestion to assist in image generation (pages 1 and 8). Wang explains its method generates a database of images and text prompts used by real users (page 1/Fig. 1). Wang further explains prompts may be suggested based on the database of images and text prompts used by other users (pages 1 and 8). It follows Feng, Brade and Bean may be accordingly modified with the teachings of Wang to select its set of text prompt suggestions based additionally on images and text prompts used by other users.
A PHOSITA before the effective filing date of the claimed invention would find it obvious to modify Feng, Brade and Bean with the teachings of Wang. Wang is analogous art in dealing with a method for prompt suggestion to assist in image generation (pages 1 and 8). Wang discloses its use of a database of images and text prompts used by real users is advantageous in appropriately suggesting prompts to assist in image generation (pages 1 and 8). Consequently, a PHOSITA would incorporate the teachings of Wang into Feng, Brade and Bean for appropriately suggesting prompts to assist in image generation. Therefore, claim 7 is rendered obvious to a PHOSITA before the effective filing date of the claimed invention.
For claim 14, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 7. It follows claim 14 is rejected for the same reasons as to claim 8 and claim 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES TSENG whose telephone number is (571)270-3857. The examiner can normally be reached 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES TSENG/Primary Examiner, Art Unit 2613