Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Claim Objections
The claim objections are withdrawn.
35 U.S.C. 112 Rejection
The previous ground of rejection under 35 U.S.C. 112 is withdrawn.
However, a new ground of rejection under 35 U.S.C. 112 for the recitation of new matter is raised to the claims as set forth below in the Detailed Action.
35 U.S.C. 103 Rejection
Applicant's arguments filed 7/06/2026 have been fully considered but they are not persuasive.
Claim(s) 1, 4-6, 8, 11-13, 15 and 18-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng et al., PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation, arXiv, August 2023 (hereinafter “Feng”) in view of Brade et al, Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models, arXiv, April 2023 (hereinafter “Brade”) (additional pages of Brade are enclosed in this Office Action and supplement the previously mailed pages of Brade enclosed with the First Office Action) and Bean (U.S. Patent Application Publication 2024/0320867 A1).
For independent claims, 1, 8 and 15, Applicants argue the references fail to disclose the newly amended limitations of “automatically displaying the set of text prompt suggestions before generating images based on the partial generative image prompt”.
However, as stated above and explained further below in the following Detailed Action, Applicants’ Specification fails to provide support for the newly amended limitations and the Examiner is unable to discern the metes and bounds of the newly amended limitations.
Nevertheless, in the interest of compact prosecution, Examiner finds the references teaches the newly amended limitations of “automatically displaying the set of text prompt suggestions before generating images based on the partial generative image prompt”.
In particular, Brade similarly discloses a method of automatic prompt suggestion to assist in image generation (page 1). Brade explains its automatic prompt suggestion automatically displays a set of text prompt suggestions before generating images based on a partial generative image prompt, e.g. automatic display of suggestions for modifying a partial generative image prompt of “lion” before generating images based on “lion” (page 1/Fig. 1; and pages 4-5/Figs. 2-3). It follows Feng may be accordingly modified with the teachings of Brade to automatically display a set of text prompt suggestions before generating images based on its partial generative image prompt and display its generated set of images on a digital canvas.
Examiner disagrees with the Applicants’ assertion that any proposed modification of Feng in this manner would change the principle of operation of Feng.
Such a modification would merely allow the user to view prompt suggestions and modify an initial prompt to appropriately refine the initial prompt prior to image generation as taught in Brade (see Brade at page 1/Fig. 1 and pages 4-5/Fig. 2).
Therefore, Examiner finds the references disclose the limitations of claims 1, 8 and 15.
For dependent claims 4-7, 11-14, and 18-20, Applicants argue for their allowanced based on their dependence to claims 1, 8 and 15. It follows claims 4-7, 11-14, and 18-20 remain rejected for the reasons discussed above and in the following Detailed Action.
For new claims 21-26, Examiner similarly finds the Specification fails to provide support for the subject matter of claims 21-26 and constitute new matter as discussed below in the following Detailed Action.
Nevertheless, Examiner finds the references disclose the subject matter of claims 21-26 for the reasons discussed below in the following Detailed Action.
DETAILED ACTION
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 4-8, 11-15 and 18-26 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
For independent claims 1, 8 and 15, these claims have been amended to recite “automatically displaying the set of text prompt suggestions before generating images based on the partial generative image prompt”. However, Applicants have failed to cite any portions of the Specification as providing support for the amended limitations. Examiner has reviewed the Specification as originally filed and has only found several instances in regards to “automatically” and “automation” at paragraphs 111-112 and 114-16. However, these passages merely discuss automation with regards to collage generation and mood generation and do not provide support for “automatically displaying the set of text prompt suggestions before generating images based on the partial generative image prompt”. Therefore, Examiner finds the Specification fails to provide support for the amended subject matter of claims 1, 8 and 15 and constitutes new matter failing to comply with the written description requirement.
Dependent claims 4-7, 11-14, and 18-26 respectively depend from independent claims 1, 8 and 15 and respectively inherit the deficiencies of independent claims 1, 8 and 15. Therefore, claims 4-7, 11-14, and 18-26 are likewise rejected under 35 U.S.C. 112 for reciting new matter.
Furthermore, for dependent claims 7 and 14, these claims have been amended to recite “one or more other users sharing a characteristic with the user”. However, Applicants have failed to cite any portions of the Specification as providing support for the amended limitations. Examiner has reviewed the Specification as originally filed and has only found one instance regarding sharing of characteristics at paragraph 113. However, this passage discusses sharing of characteristics between seed images and does not provide support for “one or more other users sharing a characteristic with the user”. Therefore, Examiner finds the Specification fails to provide support for the amended subject matter of claims 7 and 14 and constitutes new matter failing to comply with the written description requirement.
Furthermore, for dependent claims 21, 23 and 25, these claims have been amended to recite “one or more candidate completions”. However, Applicants have failed to cite any portions of the Specification as providing support for the amended limitations. Examiner has reviewed the Specification as originally filed and has not found any instance of “candidate” in the Specification. Therefore, Examiner finds the Specification fails to provide support for the amended subject matter of claims 21, 23, and 25 and constitutes new matter failing to comply with the written description requirement.
Furthermore, for dependent claims 22, 24 and 26, these claims have been amended to recite “displaying, within the text entry interface element, the second set of text prompt suggestions before generating images based on the modified partial generative image prompt”. However, Applicants have failed to cite any portions of the Specification as providing support for the amended limitations. Examiner has reviewed the Specification as originally filed and it appears paragraph 93 discusses the presentation of a second set of text prompt suggestions. However, paragraph 93 discloses: For instance, if the user selects the second suggested text prompt “a big red barn overgrown with lush ivy in the early morning fog” from the second set of text prompt suggestions 1414 of FIG. 14B, then the previously modified set of images of big red barns overgrown with ivy can be further modified by the generative design system 300 to include fog in an early morning setting (for instance by applying one or more generative content operations on the set of images to include content representative of fog in the early morning) (emphasis added). Thus, paragraph 93 of the Specification appears to disclose the second set of text prompt suggestions is displayed after generating images based on the modified partial generative image prompt and not before generating images based on the modified partial generative image prompt. Examiner finds the Specification fails to provide support for the amended subject matter of claims 22, 24 and 26 and constitutes new matter failing to comply with the written description requirement.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-6, 8, 11-13, 15 and 18-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng et al., PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation, arXiv, August 2023 (hereinafter “Feng”) in view of Brade et al, Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models, arXiv, April 2023 (hereinafter “Brade”) (additional pages of Brade are enclosed in this Office Action and supplement the previously mailed pages of Brade enclosed with the First Office Action) and Bean (U.S. Patent Application Publication 2024/0320867 A1).
For claim 1, Feng discloses a method (pages 1 and 9) comprising: receiving, within a text entry interface element of an image generation system, a partial generative image prompt from a user (disclosing an image generation system (page 4) for presenting a user interface with a text entry interface element to receive a first prompt as a partial generative image prompt from a user (page 6/Fig. 4)); selecting, by the image generation system, a set of text prompt suggestions to present to the user based on the partial generative image prompt, each text prompt suggestion corresponding to a characteristic of an image to be generated based on the partial generative image prompt (disclosing the image generation system selects text prompt suggestions for presentation to the user based on the first prompt where each suggestion corresponds to a characteristic of an image to be generated based on the first prompt (pages 6-7/Fig. 4)); receiving, by the image generation system, a selection by the user of a text prompt suggestion from the set of text prompt suggestions (disclosing the image generation system receives a selection of a text prompt suggestion from the user from the set of text prompt suggestions (pages 6-7/Fig. 4)); modifying, by the image generation system, the partial generative image prompt using the selected text prompt suggestion to produce a modified generative image prompt (disclosing the image generation system modifies the first prompt using the selected text prompt suggestion to produce a modified generative image prompt (pages 6-7/Fig. 4)); applying, by the image generation system, a generative image model to the modified generative image prompt to produce a set of images (disclosing the image generation system applies a generative image model such as a Stable Diffusion model to the modified generative image prompt to produce a set of images (pages 6-7/Fig. 4)); and displaying, by the image generation system, the set of images (disclosing the image generation system displays the set of images within the user interface to the user (pages 6-7/Fig. 4)).
Feng does not specifically disclose automatically displaying a set of text prompt suggestions before generating images based on a partial generative image prompt and displaying generated images within a digital canvas.
However, these limitations are well-known in the art as disclosed in Brade.
Brade similarly discloses a method of automatic prompt suggestion to assist in image generation (page 1). Brade explains its automatic prompt suggestion automatically displays a set of text prompt suggestions before generating images based on a partial generative image prompt, e.g. automatic display of suggestions for modifying a partial generative image prompt of “lion” before generating images based on “lion” (page 1/Fig. 1; and pages 4-5/Figs. 2-3). Brade further explains it is known to display generated images on a digital canvas (pages 6-7 and 10). It follows Feng may be accordingly modified with the teachings of Brade to automatically display a set of text prompt suggestions before generating images based on its partial generative image prompt and display its generated set of images on a digital canvas.
A person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention would find it obvious to modify Feng with the teachings of Brade. Brade is analogous art in dealing with a method of automatic prompt suggestion to assist in image generation (page 1). Brade discloses its use of a digital canvas is advantageous in appropriately visualizing generated images to a user for interaction (pages 6-7 and 10). Consequently, a PHOSITA would incorporate the teachings of Brade into Feng for appropriately visualizing generated images to a user for interaction.
Examiner finds Feng and Brade discloses an image generation system for the reasons discussed above.
In any case, these limitations are well-known in the art as disclosed in Bean.
Bean similarly discloses a method of prompt suggestion to assist in image generation (par. 22). Bean explains its method may be implemented with a computer system to perform image generation as an image generation system (Fig. 9; par. 22 and 95). It follows Feng and Brade may be accordingly modified with the teachings of Bean to implement its method with an image generation system.
A PHOSITA before the effective filing date of the claimed invention would find it obvious to modify Feng and Brade with the teachings of Bean. Bean is analogous art in dealing with a method of prompt suggestion to assist in image generation (par. 22). Bean discloses its use of image generation system is advantageous in appropriately implementing steps to carry out a method to perform image generation (Fig. 9; par. 22 and 95). Consequently, a PHOSITA would incorporate the teachings of Bean into Feng and Brade for appropriately implementing steps to carry out a method to perform image generation. Therefore, claim 1 is rendered obvious to a PHOSITA before the effective filing date of the claimed invention.
For claim 4, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of images includes subject matter from the partial generative image prompt and subject matter from the selected text prompt suggestion (Feng discloses the set of images includes subject matter from the first prompt as the partial generative image prompt and subject matter from the selected text prompt suggestion (page 6/Fig. 4)).
For claim 5, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of text prompt suggestions is selected based at least in part on one or more other images generated by the user within the digital canvas (Feng discloses the set of text prompt suggestions are selected based on images generated by the user from the first prompt (page 6/Fig. 4); Brade similarly discloses a method of automatic prompt suggestion to assist in image generation (page 1); Brade explains it is known to display generated images on a digital canvas (pages 6-7 and 10); and it follows Feng may be accordingly modified with the teachings of Brade to present images generated from its first prompt on a digital canvas).
For claim 6, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of text prompt suggestions is selected based at least in part on one or more other images generated by the user during a same working session (Feng discloses the set of text prompt suggestions are selected based on images generated by the user from the first prompt during a same usage scenario as a same working session (page 6/Fig. 4)).
For claim 8, Feng as modified by Brade and Bean discloses a non-transitory computer-readable storage medium storing executable instructions that, when executed by a hardware processor (Feng discloses a method (pages 1 and 9); Bean similarly discloses a method of prompt suggestion to assist in image generation (par. 22); Bean explains its method may be implemented with a computer system to perform image generation as an image generation system (Fig. 9; par. 22 and 95); Bean explains the computer system includes memory for storing instructions for execution by a processor to perform the functions of the computer system (par. 95 and 104); and it follows Feng and Brade may be accordingly modified with the teachings of Bean to implement its method with an image generation system), cause the hardware processor to perform steps of the method of claim 1 (see above as to claim 1).
For claim 11, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 4. It follows claim 11 is rejected for the same reasons as to claim 8 and claim 4.
For claim 12, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 5. It follows claim 12 is rejected for the same reasons as to claim 8 and claim 5.
For claim 13, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 6. It follows claim 13 is rejected for the same reasons as to claim 8 and claim 6.
For claim 15, Feng as modified by Brade and Bean discloses a system comprising: a hardware processor; and a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor (Feng discloses a method (pages 1 and 9); Bean similarly discloses a method of prompt suggestion to assist in image generation (par. 22); Bean explains its method may be implemented with a computer system to perform image generation as an image generation system (Fig. 9; par. 22 and 95); Bean explains the computer system includes memory for storing instructions for execution by a processor to perform the functions of the computer system (par. 95 and 104); and it follows Feng and Brade may be accordingly modified with the teachings of Bean to implement its method with an image generation system), cause the hardware processor to perform steps of the method of claim 1 (see above as to claim 1).
For claim 18, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 4. It follows claim 18 is rejected for the same reasons as to claim 15 and claim 4.
For claim 19, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 5. It follows claim 19 is rejected for the same reasons as to claim 15 and claim 5.
For claim 20, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 6. It follows claim 20 is rejected for the same reasons as to claim 15 and claim 6.
For claim 21, depending on claim 1, Feng as modified by Brade and Bean discloses wherein the set of text prompt suggestions includes one or more candidate completions of the partial generative image prompt (Brade similarly discloses a method of automatic prompt suggestion to assist in image generation (page 1); Brade explains its prompt suggestions includes a list of prompt modifiers as candidate completions for autocompleting a current prompt as a partial generative image prompt (pages 7-8); and it follows Feng may be accordingly modified with the teachings of Brade to include one or more candidate completions of its partial generative image prompt in its set of text prompt suggestions).
For claim 22, depending on claim 1, Feng as modified by Brade and Bean discloses further comprising, after modifying the partial generative image prompt using the selected text prompt suggestion: selecting, by the image generation system, a second set of text prompt suggestions to present to the user based on the modified partial generative image prompt; and displaying, within the text entry interface element, the second set of text prompt suggestions before generating images based on the modified partial generative image prompt (Brade similarly discloses a method of automatic prompt suggestion to assist in image generation (page 1); Brade discloses its automatic prompt suggestion displays a first set of text prompt suggestions for modifying a partial generative prompt using a selected text prompt suggestion, e.g. suggestions for steering the subject matter, and displaying a second set of text prompt suggestions based on the modified partial generative prompt in a prompt writing text field as a text entry interface element before generating images based on the modified partial generative image prompt, e.g. suggestions for steering the style (page 1/Fig. 1; pages 4-5/Fig. 2-3; and pages 7-8); and it follows Feng may be accordingly modified with the teachings of Brade to select a second set of text prompt suggestions to present to its user based on its modified partial generative image prompt and displaying, within its text entry interface element, the second set of text prompt suggestions before generating image based on the modified partial generative image prompt).
For claim 23, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 21. It follows claim 23 is rejected for the same reasons as to claim 8 and claim 21.
For claim 24, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 22. It follows claim 24 is rejected for the same reasons as to claim 8 and claim 22.
For claim 25, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 21. It follows claim 25 is rejected for the same reasons as to claim 15 and claim 21.
For claim 26, depending on claim 15, this claim is a combination of the limitations of claim 15 and claim 22. It follows claim 26 is rejected for the same reasons as to claim 15 and claim 22.
Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng in view of Brade and Bean further in view of Wang et al., DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models, arXiv, July 2023 (hereinafter “Wang”) and Smetanin et al. (U.S. Patent Application Publication 2024/0296606 A1, hereinafter “Smetanin”).
For claim 7, depending on claim 1, Feng as modified by Brade and Bean does not disclose a set of text prompt suggestions are selected based on text prompts used by other users.
However, these limitations are well-known in the art as disclosed in Wang.
Wang similarly discloses a method for prompt suggestion to assist in image generation (pages 1 and 8). Wang explains its method generates a database of images and text prompts used by real users (page 1/Fig. 1). Wang further explains prompts may be suggested based on the database of images and text prompts used by other users (pages 1 and 8). It follows Feng, Brade and Bean may be accordingly modified with the teachings of Wang to select its set of text prompt suggestions based additionally on images and text prompts used by other users.
A PHOSITA before the effective filing date of the claimed invention would find it obvious to modify Feng, Brade and Bean with the teachings of Wang. Wang is analogous art in dealing with a method for prompt suggestion to assist in image generation (pages 1 and 8). Wang discloses its use of a database of images and text prompts used by real users is advantageous in appropriately suggesting prompts to assist in image generation (pages 1 and 8). Consequently, a PHOSITA would incorporate the teachings of Wang into Feng, Brade and Bean for appropriately suggesting prompts to assist in image generation.
Feng as modified by Brade, Bean and Wang does not disclose one or more users sharing a characteristic with a user.
However, these limitations are well-known in the art as disclosed in Smetanin.
Smetanin similarly discloses a system and method for suggesting a text prompt to a user for automated image generation (par. 67). Smetanin explains it system implements user profiles to establish relationships between multiple users to share settings and characteristics between users to facilitate automated image generation of content (par. 72, 141 and 178). It follows Feng, Brade, Bean and Wang may be accordingly modified with the teachings of Smetanin to determine its user sharing a characteristic with one or more users.
A PHOSITA before the effective filing date of the claimed invention would find it obvious to modify Feng, Brade, Bean and Wang with the teachings of Smetanin. Smetanin is analogous art in dealing with a system and method for suggesting a text prompt to a user for automated image generation (par. 67). Smetanin discloses its use sharing between users is advantageous in appropriately facilitating automated image generation of content (par. 178). Consequently, a PHOSITA would incorporate the teachings of Smetanin into Feng, Brade, Bean and Wang for appropriately facilitating automated image generation of content. Therefore, claim 7 is rendered obvious to a PHOSITA before the effective filing date of the claimed invention.
For claim 14, depending on claim 8, this claim is a combination of the limitations of claim 8 and claim 7. It follows claim 14 is rejected for the same reasons as to claim 8 and claim 7.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES TSENG whose telephone number is (571)270-3857. The examiner can normally be reached 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES TSENG/ Primary Examiner, Art Unit 2613
Prosecution Insights